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COST  -EFFECTIVENESS  OPTIMIZATION 
(TASKS  AND  ANALYSIS  METHODOLOGY) 


FOREWORD 


This  is  Volume  II  of  the  final  report  of  Task  Group  IV  of  the  Weapon 
System  Effectiveness  Industry  Advisory  Committee  (WSEIAC).  It  is  sub¬ 
mitted  to  the  Commander,  AFSC  in  partial  fulfillment  of  Task  Group  IV 
objectives  cited  in  the  committee  Charter.  The  final  report  is  contained  in 
three  separate  volumes: 

Volume  I  presents  a  summary  of  the  principles  of  cost- 
effectiveness  analysis,  conclusions  and  recommendations. 

Volume  II  contains  a  discussion  of  the  specific  tasks 
required  to  conduct  a  cost-effectiveness  analysis, 
emphasizing  procedural  and  analytical  techniques. 

Volume  III  consists  of  a  technical  supplement  illustrating 
some  of  the  methodology  appropriate  to  cost-effectiveness 
analysis. 
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WSEIAC  CHARTER 


In  order  that  this  report  of  Task  Group  IV  may  be  studied  in  context 
with  the  entire  committee  effort,  the  purpose  and  task  group  objectives  as 
stated  in  the  WSEIAC  Charter  are  listed  below: 

Purpose 

The  purpose  of  the  Weapon  System  Effectiveness  Industry  Advisory 
Committee  is  to  provide  technical  guidance  and  assistance  to  AFSC  in  the 
development  of  a  technique  to  apprise  management  of  current  and  predicted 
weapon  system  effectiveness  at  all  phases  of  weapon  system  life. 

Task  Group  Objectives 

Task  Group  I  -  Review  present  procedures  being  used  to  establish  system 
effectiveness  requirements  and  recommend  a  method  for  arriving  at  require¬ 
ments  that  are  mission  responsive. 

Task  Group  II  -  Review  existing  documents  and  recommend  uniform  methods 
and  procedures  to  be  applied  in  predicting  and  measuring  systems  effective¬ 
ness  during  all  phases  of  a  weapon  system  program. 

Task  Group  III  -  Review  format  and  engineering  data  content  of  existing 
system  effectiveness  reports  and  recommend  uniform  procedures  for 
periodically  reporting  weapon  system  status  to  assist  all  levels  of  manage¬ 
ment  in  arriving  at  program  decisions. 

Task  Group  IV  -  Develop  a  basic  set  of  instructions  and  procedures  for 
conducting  an  analysis  for  system  optimization  considering  effectiveness, 
time  schedules,  and  funding. 

Task  Group  V  -  Review  current  policies  and  procedures  of  other  Air  Force 
commands  and  develop  a  framework  for  standardizing  management  visibility 
procedures  throughout  all  Air  For-  e  commands. 


ABSTRACT 


This  report  discusses  the  philosophy  of  cost-effectiveness  and  techniques 
for  trade-off  and  optimization  studies.  It  lists  and  discusses  twelve  tasks 
necessary  to  perform  a  cost-effectiveness  analysis.  A  methodology  is  out¬ 
lined  for  identifying  and  standardizing  cost  and  effectiveness  factors. 
Descriptive  analytical  models  for  cost-effectiveness  are  provided,  including 
discussion  of  their  sensitivity  and  validity.  One  section  defines  and  dis¬ 
cusses  risk  and  uncertainty  and  their  effect  on  the  decision  making  process. 
Included  is  an  extensive  bibliography  on  cost-effectiveness.  Examples  of 
some  of  the  techniques  are  covered  in  detail  in  a  "Technical  Supplement," 
which  is  Volume  HI  of  this  final  report  of  Task  Group  IV.  Abstracts  of 
these  examples  will  be  found  in  Appendix  1  of  this  report.  Appendix  II  of 
this  report  illustrates  a  technique  for  cost -effectiveness  optimization  in  the 
Definition  Phase  whes;  there  is  a  relative  dearth  of  data,  program  objectives 
are  fixed,  and  system  effectiveness  is  unconstrained. 
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SECTION  I 


INTRODUCTION 

A  major  management  goal  throughout  the  life  cycle  of  a  system  --  from 
the  Conceptual  Phase  through  the  Operational  Phase  —  is  to  exercise 
management  control  for  the  purposes  of  selecting;  developing,  and  using 
systems  in  an  optimum  manner.  The  process  by  which  management  is 
provided  inputs  for  these  types  of  decisions  has  been  commonly  called  cost- 
effectiveness  analysis. 

Effectiveness  is  a  measure  of  the  capability  of  the  system  to  accomplish 
the  mission  objectives.  Cost-effectiveness  studies  are  concerned  with 
achieving  a  combination  of  resource-use  and  attained  effectiveness  that  is 
best  according  to  a  selected  criterion.  Resource-use  represents  the  ex¬ 
penditure  of  dollars,  manpower,  material,  time,  etc.,  required  for  the 
development,  operation,  and  support  of  a  system.  We  shall  interpret  such 
studies  as  an  attempt  to  quantify  how  much  it  costs  to  achieve  a  certain 
effectiveness  in  order  to  select  among  a  set  of  alternatives. 

There  is  a  recognised  need  for  such  studies.  The  enormous  respon¬ 
sibility  of  the  Department  of  Defense  and  the  military  services  for  main¬ 
taining  a  strong  posture  involves  considerable  expenditure  of  national 
resources.  This  is  clearly  evidenced  by  the  proportion  of  the  federal 
budget  now  allocated  to  defense.  It  is  thus  mandatory  that  the  military 
authorities  exercise  maximum  control  in  their  planning,  procurement, 
and  operational  activities  in  order  tc  minimise  the  burden  placed  on  the 
economy  without  any  sacrifice  in  over -all  defense  goals. 

Cost-effectiveness  analysis  is  not  new.  It  has  been  a  part  of  military 
planning  for  some  time,  but  the  complexity  of  the  military  tasks  now 
require  a  multidisciplinary  apprcach.  The  major  utility  of  cost-effectiveness 
analysis  is  to  provide  management  with  the  necessary  information  for 
decision  making  purposes  utilising  all  the  available  knowledge  and  data  in 
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as  efficient  and  complete  a  manner  as  is  possible.  Consequently,  a  demand 
has  been  created  for  improved  analytical  methods,  better  and  more  complete 
data,  expanded  computational  capacity,  etc.,  which  has  improved  and  will 
continue  to  improve  management's  capability  for  making  good  decisions. 

A.  LEVELS  OF  COST-EFFECTIVENESS  ANALYSES 

There  are  several  decision  making  levels  at  which  a  cost-effectiveness 
analysis  can  be  meaningfully  applied,  and  these  roughly  correspond  to  the 
phases  during  system  development.  One  level  for  application  is  at  the 
Required  Operational  Capability  (ROC)  level,  formerly  called  the  General 
Operational  Requirement  (GOR)  phase.  The  ROC  establishes  a  spectrum  of 
objectives  or  missions.  By  considering  over -all  defense  goals,  the  geo¬ 
political  and  environmental  factors,  and  the  economic  and  technological 
capabilities,  a  particular  mission  or  objective  is  selected.  This  level  of 
application  is  generally  coordinated  at  the  DOD  level. 

After  mission  requirements  are  set  at  the  Specific  Operational 
Requirements  (SOR)  phase,  there  exists  the  need  for  selecting  alternate  or 
competing  systems.  Application  of  cost-effectiveness  analysis  at  this  level 
is  primarily  the  responsibility  of  the  military  or  procuring  agencies. 

A  third  level  of  application  occurs  during  the  development  and  opera¬ 
tion  of  the  weapon  system.  This  level  of  application  furnishes  information 
for  optimal  use  of  resources  within  the  constraints  of  mission  and  system 
requirements. 

\s  a  simple  example  of  these  levels,  the  first  would  be  concerned  with 
such  problems  as  optimum  force  mix;  e.g.,  expanded  bomber-force  size 
versus  expanded  missile-force  size.  The  second  level  would  be  concerned 
with  such  combinatorial  choices  as  pertain  within  a  class  of  systems;  e.g., 
within  missile  systems  we  may  examine  liquid  versus  solid  fuel,  tandem 
versus  parallel  stages,  or  soft  versus  hardened  sites.  The  third  level 
would  be  concerned  with  more  detailed  decisions  within  a  given  system  con¬ 
figuration;  e.g.,  for  a  missile  one  might  evaluate  pressurized  or  pump-fed 
propellant  loading  systems,  various  stage  diameters,  various  area  ratios 
of  engine  nozzles,  checkout  and  monitoring  procedures,  and  the  like. 


This  report  is  concerned  primarily  with  the  third  level  of  analysis,  and 
to  a  lesser  extent  with  the  second  level,  in  presenting  and  illustrating  the 
concepts,  methods,  and  procedures  of  cost-effectiveness  analysis. 

B.  GENERAL  CONCEPTS 

To  introduce  the  general  concepts  of  a  cost-effectiveness  analysis,  we 
shall  interpret  such  analysis  in  the  simplest  of  terms  --  namely,  the  attempt 
to  quantify  how  much  it  costs  to  achieve  a  certain  effectiveness  in  order  to 
select  among  a  set  of  alternatives.  Cost  is  used  to  represent  the  amount  of 
resource  expenditure,  and  effectiveness  is  a  measure  of  the  system  ability 
to  accomplish  its  mission  objectives. 

The  general  approach  for  making  such  decisions  consists  of  the 
following  steps: 

(1)  Define  criterion  for  selection 

(2)  Generate  alternatives  that  satisfy  operational  requirements 
and  constraints 

(3)  Compute  resultant  values  of  cost  and  effectiveness  for  each 
alternative 

(4)  Evaluate  results  with  respect  to  the  decision  criterion. 

Each  of  these  major  steps  is  discussed  in  detail  in  the  report.  It  is 
worthwhile,  however,  to  set  the  stage  for  such  discussions  in  this  intro¬ 
duction. 

The  criterion  for  selection  must  be  one  that  is  mission  responsive; 
that  is,  it  must  answer  the  right  question.  Essentially,  the  criterion  is 
based  on  maximizing  effectiveness  for  a  given  cost  or,  conversely,  mini¬ 
mizing  cost  for  a  given  level  of  effectiveness.  The  criterion,  however, 
must  also  define  the  level  of  analysis  as  discussed  previously  in  this  intro¬ 
duction  and  also  the  scope  of  the  analysis  in  terms  of  resource,  system, 
operational  and  support  constra.nts.  Thus,  the  foi  r  basic  criteria  listed 
above  may  evolve  into  a  criterion  such  as  one  to  maximize  effectivenesc  per 
dollar,  provided  effectiveness  is  greater  than  E*  and  cost  is  less  than  C* 
(where  E*  and  C*  refer  to  specific  limiting  values). 


In  generating  acceptable  alternatives,  identification  of  all  variable  and 
fixed  factors  and  their  costs  is  required.  In  addition,  the  elements  of  risk 
and  uncertainty  as  related  to  these  factors  and  costs  and  the  analysis  of 
effects  on  other  programs  must  also  be  considered.  Such  factors  as  avail¬ 
ability  of  appropriate  data,  computational  capacity,  and  restraints  in  time 
and  effort  available  for  the  analysis  will  play  important  roles  in  this  phare. 

A  generated  alternative  is  then  an  acceptable  combination  of  the  selected 
factors  with  associated  risk  and  uncertainty  elements. 

Measures  of  cost  and  effectiveness  for  each  design  alternative  must 
then  be  computed.  The  form  these  measures  take  is  related  to  the  decision 
criterion.  For  effectiveness,  the  measure  can  range  from  a  simple  proba¬ 
bility  numeric,  to  an  expected  value,  to  the  complete  distribution  of  some 
over-all  performance  characteristic.  The  effectiveness  model  is  based  on 
sub-models  for  reliability,  maintainability,  and  performance.  These  in 
turn  are  based  on  the  variable  and  fixed  factors  to  be  considered  such  as 
failure  and  repair  distributions,  internal  stresses,  environment,  and  design 
integration. 

The  cost  measure  must  be  one  that  can  treat  the  major  types  of  resource 
expenditures  on  some  common  basis.  Sub-models  are  required  for  develop¬ 
ment  costs,  operating  costs,  and  support  costs  both  in  terms  of  dollars  and 
schedules.  In  addition,  the  burden  a  particular  alternative  places  on  other 
systems  and  objectives  must  be  evaluated  for  a  complete  cost  model. 

The  integration  of  the  separate  cost  and  effectiveness  models  into  a 
single  cost-effectiveness  model  provides  the  basis  for  decisions.  It  is  at 
this  stage  where  optimisation  theory  becomes  applicable,  involving  such 
disciplines  as  mathematical  programming,  stochastic  process  theory,  cal¬ 
culus  of  variations,  econometrics,  ind  decision  theory. 

All  of  the  above  models  must  satisfy  characteristics  related  to  adequacy, 
representativeness,  consistency,  sensitivity,  plausibility,  criticality,  work¬ 
ability,  and  suitability.  These  characteristics  are  discussed  more  fully  in 
later  sections.  In  applying  the  moael,  it  must  be  emphasised  that  results 
of  the  optimisation  process  can  only  indicate  the  best  decision  within  the 
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simplification!,  assumptions,  restrictions  and  omission  that  were  required 
to  circumvent  such  problems  as  uncertainties,  nonquanti Sable  factors,  and 
inadequate  data,  time  or  computational  capacity. 

Thus,  the  cost-effectiveness  analysis  will  usually  yield  only  partial 
analytic  solutions.  However,  the  framework  for  a  final  decision  is  provided. 
The  cost-effectiveness  analysis  has  reduced  the  guess  work  and  intuitive 
estimates  of  cost  and  effectiveness,  but  the  initial  results  must  still  be 
critically  evaluated  and  combined  with  relevant  political  and  timing  factors 
by  a  judgment  of  the  decision  maker. 
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SECTION  II 


DISCUSSION  OF  TASKS  AND  FACTORS  IN 
COST-EFFECTIVENESS  ANALYSIS 

There  is  no  unique  approach  to  performing  a  cost-effectiveness  study 
just  as  there  is  no  single  detailed  model  which  will  be  useful  in  studying  all 
systems.  A  cost-effectiveness  study  may  be  regarded  as  one  form  of  system 
analysis.  In  general,  certain  tasks  and  input  elements  are  involved  in  such 
analyses.  Figure  1  provides  a  typical  example  of  tasks  and  their  sequential 
relationship.  These  tasks  and  input  elements  will  be  briefly  described  here 
in  order  to  orient  the  reader  to  the  over -all  cost-effectiveness  optimisation 
process.  Discussion  of  methods  associated  with  the  tasks  will  be  covered  in 
subsequent  sections  of  this  document. 

A.  TASKS 

1.  Define  Program  Objectives  and  Mission  The  introduction  pointed 
out  that  cost-effectiveness  studies  are  conducted  at  three  basic  levels; 
selection  of  mission,  selection  of  system,  and  optimisation  of  system.  The 
first  or  highest  level  of  these  studies  is  not  within  the  scope  of  this  report. 
However,  the  results  of  cost-effectiveness  studies  at  this  higher  level  must 
provide  the  initial  basis  for  establishing  program  objectives  and  require¬ 
ments  to  permit  the  lower  level  optimisations.  It  is  fundamental  to  the 
basic  definition  of  effectiveness  that  the  end  goals  and  purpose  be  defined 
so  that  a  system  can  be  optimised  in  terms  of  a  specified  mission  or  task  to 
be  accomplished. 

2.  Identify  Resources  and  Constraints  In  addition  to  stipulating  the 
program  task,  it  is  necessary  to  define  resources  and  constraints.  Re¬ 
sources  are  those  items  (i.  e. ,  people,  technology,  dollars,  etc. )  available 
to  accomplish  a  task  or  program.  Constraints  are  limits  on  resources.  The 
resources  and  constraints  are  inherent  in,  or  a  direct  result  of,  the  state¬ 
ment  of  a  program  task  or  a  program  definition.  Although  it  is  possible  to 
design  a  system  without  prior  knowledge  of  the  resources  which  are  avail¬ 
able,  it  is  not  logical  or  usual  to  do  so.  The  optimixation  of  a  system  to 
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provide  greatest  effectiveness  in  accomplishing  the  required  task  per 
amount  of  resource  expenditure  requires  an  even  more  detailed  knowledge 
of  the  available  resources,  together  with  the  relationship  of  resources  to 
alternative  means  of  meeting  the  stated  task. 

3.  Selection  of  Cost-Effectiveness  Criteria  and  Measures  The 
optimization  process  is  essentially  one  of  achieving  a  combination  of  re- 
source  use  and  attained  effectiveness  that  is  best  by  some  criterion.  In 
order  to  render  the  optimization  process  feasible,  a  criterion  must  be  de¬ 
veloped  that  is  expressed  in  precise,  quantifiable  terms.  These  criteria 
are  developed  in  recognition  of  the  stated  program  objectives  and  those 
limitations  inherent  in  the  available  resources. 

The  quantitative  terms  or  measures  generally  associated  with  the 
effectiveness  criteria  are  usually  expressed  in  terms  of  cost,  which  gen¬ 
erally  represents  expenditures  not  only  of  dollars  but  also  of  time,  man¬ 
power,  and  other  resources. 

4.  Identify  and  Synthesize  Alternatives  The  ability  to  optimize  a 
system  depends  on  the  availability  of  alternate  means  of  meeting  the 
requirements.  Alternatives  include  the  means,  approaches,  or  techniques 
which  can  be  employed  to  meet  the  program  objectives  and  missions  within 
the  constraints  of  the  resources.  As  can  be  seen  from  Figure  1  the 
optimization  approach  is  generally  based  on  a  fundamental  sequence  of: 

a.  Identification  of  alternatives 

b.  Evaluation  of  alternatives 

c.  Selection  of  that  alternative  or  combination 

of  alternatives  which  provides  the  most 

cost-effective  system. 

The  task  of  identifying  alternatives  consists  basically  of  examining 
the  objectives  and  missions  together  with  the  basic  system  concept  proposed 
and  listing  all  possible  alternative  means  of  meeting  the  objectives  The 
alternatives  thus  listed  should  then  be  screened  against  the  available  re¬ 
sources  and  stated  bounds  to  insure  that  they  are  in  fact  feasible. 

Synthesis  of  alternatives  consists  of  those  steps  needed  to  defini¬ 
tive  the  alt  ernate  design  c  onfigurat  ions .  operational  plans,  etc  associated 
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with  listed  Alternatives  to  that  degree  of  detail  needed  to  permit  their 
subsequent  evaluation. 

5.  Identify  Variables  A  variable  is  defined  here  as  a  parameter  or 
quantity  the  use  of  which,  when  varied,  will  result  in  variations  in  re¬ 
sources  or  the  effectiveness  with  which  the  program  objectives  are 
accomplished.  This  step  in  the  optimisation  process  consists  of  identifying 
those  variables  which  will  influence  the  evaluation  of  each  alternative  listed 
from  the  preceeding  task'. 

6.  Develop  Relationships  and  Models  The  next  step  in  the  basic 
optimisation  process  is  to  relate  the  variables  used  with  each  other  and  with 
the  resources  which  are  affected.  These  relationships  must  be  expressed 

in  such  a  manner  that  the  variables  and  resources  can  be  expressed  in  terms 
of  the  established  cost-effectiveness  criteria  and  measures.  Development 
of  relationships  is  carried  to  the  point  where  all  resources  and  variables 
can  eventually  be  related  to  either  a  single  common  denominator,  (usually 
dollars),  or  to  a  cost  denominator  (dollars)  and  an  effectiveness  measure. 
The  relationships  so  developed  are  then  expressed  in  n,odel  form,  which  is 
essentially  a  mathematical,  logical,  or  physical  representation  of  the  inter¬ 
dependencies  between  the  variables,  resources,  and  measures  of  effective¬ 
ness. 

7.  Validation  and  Sensitivity  of  Model  It  must  be  recognised  that  the 
models  developed  above  will  not  be  exact  replicas  of  the  "real  world.  " 
Accordingly,  they  should  not  be  used  blindly.  Portions  of  every  model  are 
usually  common  to  previously  used  models  or  can  be  related  to  quantitative 
knowledge  of  trends  available  from  past  experience.  The  model  is  validated 
by  checks  in  as  many  familiar  regions  as  possible.  The  model  is  also 
checked  for  sensitivity  of  its  output  to  changes  in  its  basic  structure.  These 
sensitivity  checks  are  made  in  all  areas  where  simplifications  have  been 
made  from  the  "real  world"  case  or  where  anomalies  have  resulted  from 
the  validation  checks. 

8.  Develop  Inputs  This  step  requires  no  description  other  than  to 
indicate  that  it  must  include  documentation  of  all  input  data,  its  associated 
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range  of  uncertainty,  and  other  validating  information.  Inputs  are  re¬ 
quired  in  area*  of  cost  of  vehicles,  bases,  etc.,  effectiveness  values  such 
as  kill  probabilities,  readiness,  probabilities,  etc.,  and  estimates  of  risk 
probabilities. 

9.  Evaluate  Alternatives  This  step  consists  of  application  of  the 
developed  models  and  inpuC  data  to  each  alternative.  This  task  may  be 
accomplished  in  several  increments,  starting  with  evaluation  of  alternatives 
on  an  individual  basis  ending  with  a  series  of  iterations  or  simulation  of 
the  complete  system  based  on  the  individual  analysis  results.  This  task 
may  also  be  accomplished  as  a  single  step  by  application  of  a  complete  sys¬ 
tem  model  or  simulation  technique. 

10.  Interpretation  and  Sensitivity  of  Results  During  and  after  the 
accomplishment  of  the  cost-effectiveness  analysis,  the  results  of  the  study 
must  be  interpreted  in  terms  useful  for  the  decision  process.  Of  particular 
importance  is  the  sensitivity  of  the  results  (i.  e.,  in  terms  of  cost- 
effectiveness  measures)  to  variations  in  the  input  data.  Thus,  if  the  cost- 
effectiveness  measure  varies  greatly  with  some  design  parameters,  the 
decision  process  must  consider  carefully  the  uncertainty  and  the  price  paid 
by  failure  to  achieve  a  design  goal. 

Interpretation  is  particularly  difficult  due  to  the  usual  communi¬ 
cation  problems  among  people  of  differing  backgrounds  and  interests.  This 
difficulty  is  amplified  by  the  nature  of  qualifying  statements  which  must  be 
made  concerning  cost-effectiveness  results  due  to  risk  and  uncertainty  and 
related  result  sensitivity. 

11.  Decisions  It  should  be  stated  again  that  the  executive,  not  the 
analyst  cr  the  CE  study,  makes  the  decision.  The  study  helps  eliminate 
uncertainty  just  as  performance  data  helps  the  engineer  select  a  component 
for  a  system.  Other  factors  must  enter  into  the  decision  process. 

12.  Implementation  of  Decisions  It  is  not  enough  fcr  the  executive  to 
announce  his  decision.  The  total  study  program  has  failed  if  proper  steps 
for  implementing  the  decision  are  not  devised  and  enacted. 
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B.  FACTORS 


There  can  be  nu  other  point  of  departure  in  a  systems  cost  effective* 
ness  study  than  an  understanding  of  the  physical  characteristics  of  the  air 
fratrs  (physical  system),  the  equipment  within  the  air  frame,  the  ground- 
support  equipment,  the  interrelation  among  individuals  and  the  various 
subsystems,  and  the  etiect  of  enemy  action  and  technology.  This  is  best 
indicated  schematically  by  Figure  2.  Development  of  the  understanding  to 
an  adequate  level  of  detail  will  in  turn  permit  the  analyst  to  reduce  these 
characteristics  into  the  factors  which  relate  to  cost  effectiveness.  The 
basic  factors  which  enter  into  a  cost-effectiveness  optimization  analysis 
may  be  categorized  as: 

Effectiveness  factors; 

Cost  factors;  and 

Risk  and  Uncertainty  factors. 

The  term  factors,  as  referred  to  in  this  discussion,  is  defined  as  those 
quantitative  terms  which  enter  into  the  models  and/or  equations  by  which 
cost-effectiveness  is  quantified  and  optimized.  The  typi'-al  factors  within 
these  categories  are  listed  in  Table  I.  They  are  further  defined  in 
succeeding  sections  of  this  report. 


THE  COST-EFFECTIVENESS  STUDY  PROCESS 


SECTION  m 


INPUTS  AND  RELATIONSHIPS^ 

(TASKS  1-5) 

This  section  discusses  the  first  five  tasks  in  Figure  1  as  they  apply  to 
second  and  third  level  analysis. 

In  the  introduction,  it  was  indicated  that  there  are  three  basic  identifi¬ 
able  levels  of  cost-effectiveness  analyses.  They  are: 

1.  Selection  of  mission 

2.  Given  mis  sion-- selection  of  competing  systems 

3.  Given  system- -selection  of  optimal  resource  use. 

Also  stated  in  the  introduction,  the  first  level  analysis  is  a  Department 
of  Defense  level  function  and  is  not  within  the  scope  of  this  report.  It  is, 
however,  necessary  to  recognise  that  these  levels  of  analysis  must  be  co¬ 
ordinated.  Figure  3  depicts  the  relationship  of  the  analyses  levels  and  the 
types  of  information  required  for  integration. 

A.  DEFINE  PROGRAM  OBJECTIVES.MISSION  AND 

CONSTRAINTS  (1ASKS  l  and  i) 

The  output  of  the  mission  selection  analyses  conducted  at  DOD  level  is 
a  statement  of  what  the  program  is  *o  accomplish.  Historically,  this  input 
covers  the  end-item  functions  which  are  to  be  accomplished  (target  destruc¬ 
tion,  reconnaissance^space  exploration,  etc.  )  and  the  conditions  and  geo¬ 
graphic  locations  within  which  these  functions  are  to  take  place.  The  opti¬ 
misation  processes  that  take  place  during  the  lower  level  cost-effectiveness 
studies  consist  mainly  of  synthesising  alternate  means  of  meeting  stated 
objectives,  evaluating  them,  and  (electing  thr  combination  of  such  alterna¬ 
tives  which  secures  the  most  favorable  cost  and  effectiveness  relationship. 
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It  follows  that  if  the  statement  of  program  objectives  severely  limits  the 
alternatives  which  can  be  considered,  the  ability  to  achieve  optimum  cost- 
effectiveness  during  the  lower  level  analyses  is  correspondingly  reduced. 
Thus,  the  statement  of  program  objectives  and  mission  derived  from  the 
first  level  analyses  should  define  what  is  to  be  done  rather  than  how  the  task 
is  to  be  accomplished.  It  is  recognized,  however,  that  additional  constraint 
and  relationship  data  must  be  provided  in  order  that  system  and  resource- 
use  selection  correspond  to  the  basis  on  which  the  mission  was  initially 
justified. 

In  order  for  a  mission  to  have  been  selected,  it  must  have  been  pre¬ 
viously  justified  on  the  basis  of  a  grossly  estimated  set  of  effectiveness  and 
cost  figures.  Operational  data  and  system  life  must  also  have  entered  into 
the  justification.  Alternatively,  a  mission  may  be  selected  contingent  upon 
an  ability  to  accomplish  it  within  a  given  cost  or  above  a  minimum  level  of 
effectiveness.  The  point  here  is,  once  a  mission  has  been  selected  and 
justified  by  the  DOD-level  analyses,  gross  economic  value  and  gross 
effectiveness  measures  must  have  been  evaluated  and  judged  acceptable.  No 
doubt  there  also  exist  limits  on  maximum  cost,  minimum  effectiveness,  or 
combinations  thereof  beyond  which  the  selected  mission  is  no  longer 
justified. 

Accordingly,  during  the  mission  selection  phase  it  is  possible  to  develop, 
from  this  inherent  or  implicit  justifying  data,  the  information  on  constraints 
and  relationships  which  is  required  to  conduct  lower  level  optimizations. 
Typical  types  of  such  data  are: 

(1)  maximum  total  cost  beyond  which  the  mission  is  not 
justified; 

(2)  minimum  effectiveness  level  below  which  the  mission 

is  not  justified,  together  with  the  measures  used  to  assess 
effectiveness  level; 

(3)  calendar  time  phasing  associated  with  the  above  values 
(this  variation  generally  results  from  the  availability  of 
competing  or  complementing  missions); 

(4)  limitations  on  or  values  associated  with  resources  to 
be  expended  in  accomplishment  of  the  mission; 
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(5)  maximum  acceptable  level  of  risk  of  failure  to  complete 
program  objectives,  or  a  dollar  value  associated  with 
varying  degiees  of  risk. 

Data  of  this  type  would  provide  the  necessary  constraints  in  a  manner  or 
form  that  assures  minimum  exclusion  of  possible  alternatives,  ft  should  be 
noted  that  all  of  the  data  types  listed  need  not  be  furnished  for  a  selected 
mission  since  the  list  includes  alternative  means  of  stating  the  same  basic 
constraint. 

B.  CRITERIA  SELECTION  (TASK  3) 

A  cost-effectiveness  optimisation  process  is  essentially  one  of  achieving 
a  combination  of  resources  and  attained  effectiveness  that  is  best  by  some 
criterion.  In  defining  an  optimising  criterion,  the  system  analyst  is  faced 
with  a  problem  similar  to  that  of  stating  in  precise,  quantifiable  terms  the 
rules  or  criteria  for  choosing  the  "best*  painting  or  "best"  automobile. 

These  examples  do  have  some  quantifiable  (though  not  necessarily  pertinent) 
characteristics,  such  as  the  sise  of  the  painting,  rating  of  the  artist,  or  the 
dimensions  (roominess)  of  the  automobile;  however,  artistic  judgment  and 
user  experience,  respectively,  are  also  factors  in  the  final  choice,  hi  the 
same  sense,  the  choice  of  the  best  weapon  system  is  greatly  influenced  by 
the  use  of  good  engineering,  economic,  and  operational  judgment. 

It  is  moot  important,  however,  that  the  optimising  criterion  which  in¬ 
cludes  as  many  significant  factors  as  possible,  be  defined  for  the  following 

reasons: 

(1)  The  inputs  provided  to  the  analyst  through  use  of  the  criterion 
can  reduce  the  sise  of  the  problem  to  a  point  where  a  judicious  choice  can  be 
made. 

(2)  Defining  a  criterion  forces  thr  analyst  to  examine  aU  possible 
alternatives  in  an  objective  manner  so  that  the  criterion  can  be  adapted  to 
mathematical  representation  and  analysis. 

(3)  It  is  easier  to  incorporate  the  ideas  and  experience  of  others  if 
a  formal  basis  for  optimisation  is  established. 


4.  The  (partial)  basis  for  final  choice  is  in  precise,  quantifiable 
terms  and  can,  therefore,  be  reviewed  and  revised,  and  can  provide  inputs 
to  a  learning  process  for  future  optimization  problems. 

When  a  criterion  for  optimization  is  being  formulated,  the  system  and 
the  boundaries  must  be  explicitly  defined.  This  definition  will  influence  the 
choice  of  parameters  in  the  optimization  model.  The  purchaser  of  a  new 
automobile,  for  example,  may  or  may  not  consider  the  service  policies  of 
the  marufacturer  and  dealer.  If  he  does,  the  system  is  both  the  automobile 
and  service  policies;  if  he  does  not,  the  system  is  only  the  automobile.  In 
attempting  to  optimize  a  weapon  system  such  as  &  bomber,  the  analyst  has 
to  consider  whether  the  system  is  to  be  defined  as  a  single  bomber,  a 
squadron  of  bombers,  or  the  complete  bomber  fleet.  It  is  possible  that 
optimizing  with  respect  to  a  single  bomber  (a  sub-optimization)  may  not 
yield  the  optimum  "squadron"  system,  which  may  not,  in  turn,  give  a  force¬ 
wide  optimum.  (See  reference  2). 

As  part  of  the  system-definition  process,  the  analyst  also  determiner 
the  fixed  and  variable  factors  pertinent  to  the  system.  This  task  requires 
a  preliminary  analysis,  since  consideration  of  all  possible  alternatives  wi  1 
usually  lead  to  problems  of  unmanageable  size.  Some  factors  may  be  con¬ 
sidered  fixed  if  results  of  previous  analyses,  perhaps  sub-optimizations, 
indicate  the  values  that  have  attained  the  best  results  in  the  past.  The 
maintenance  trouble-shooting  routine,  for  example,  might  normally  be  con¬ 
sidered  as  a  variable  factor,  but  past  research  in  this  area  may  be  used  to 
select  a  particular  routine  applicable  to  the  system  under  study,  or  perhaps 
to  restrict  the  range  to  several  alternatives. 

Once  the  mission  profile  is  defined,  consideration  can  be  given  to  the 
physical  and  economic  limitations  that  will  have  to  be  imposed.  These 
limitations  are  based  on  requirements  and  availabilities,  and  may  involve 
such  factors  as  minimum  system  output,  minimum  reliability,  minimum 
maintainability,  maximum  development  time,  maximum  weight  and  volume, 
and  type  and  nufnber  of  support  and  operational  personnel.  Through  such 
consideration  an  envelope  of  design,  development,  operational  and  support 
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alternatives  can  be  established  in  such  a  way  that  each  over -all 
configuration  within  the  envelope  will  meet  physical  and  economic  limita¬ 
tions  as  well  as  minimum  performance  goals. 

Now  the  analyst  must  select  a  decision  criterion  by  specifying  the  types 
of  effectiveness  and  cost  parameters  to  be  investigated  and  by  assigning 
numerical  values  where  required.  For  this  purpose  the  effectiveness 
measures  defined  should  be  as  precise  as  possible.  They  must  also  pro¬ 
vide  an  understandable  and  calculable  measure  of  the  system's  capability 
for  successful  accomplishment  of  mission  objectives.  As  suggested  above, 
they  should  also  include  as  many  significant  factors  as  possible.  The 
choice  of  objectives  and  criteria  is  perhaps  the  most  difficult  task  in  sys¬ 
tem  effectiveness  optimisation.  It  is  expected,  however,  that  current 
effort  in  the  optimising  cf  system  effectiveness  will  develop  theory  and 
accumulate  experience  to  help  overcome  some  of  the  difficulties  of  this 
task. 

It  is  impossible  to  establish  rigid  ground  rules  or  procedures  foi  for¬ 
mulating  a  criterion  for  optimising  Cost-Effectiveness  of  a  weapon  system. 
The  answers  to  the  following  two  basic  questions,  however,  will  provide  i 
great  deal  of  insight  for  such  formulation: 

1.  Why  is  the  system  being  developed? 

2.  What  physical  and  economic  limitations  exist  ? 

The  answer  to  the  first  question  essentially  defines  the  mission  profile 
of  the  system.  Where  possible,  the  definition  should  be  translated  into 
quantitative  parameterc--a  difficult  task  in  many  cases.  A  performance 
measure  such  as  kill-probability  for  a  SAC  bomber  may  be  assignable,  but 
the  bomber  may  also  have  a  mission  to  act  as  a  deterrent--a  measure  that 
is  difficult,  if  not  impossible,  to  quantify.  It  is  for  this  type  of  multi¬ 
mission  case  that  judgment  will  become  especially  important.  Even  if 
quantitative  requirements  can  be  placed  on  all  mission  types,  weighting 
factors  would  have  to  be  introduced  to  quantify  the  relative  importance  of 
each  mission.  However,  when  it  is  possible  to  quantify  the  factors  involved, 
one  might  evaluate  the  effectiveness  of  a  multiple  mission  system  from: 
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where  the  subscript  i  indicate*  miaaion  type, and  other  factora  are: 

T  -  fractional  application  to  miaaion  i 

I.  -  a  meaaure  of  the  relative  importance  of  miaaion  i 

E  -  effectiveneaa  of  the  ayatem  in  accompliahing  the 
objectivea  of  miaaion  i 

Factora  that  have  relatively  little  imnact  on  over-all  effectiveneaa  or 
coat  can  be  conaidered  to  be  fixed  or,  possibly,  can  be  ignored.  There  ia, 
of  course,  a  riak  involved  if  factora  choaen  to  be  fixed  or  unimportant 
would  have  had  a  aignificant  effect  if  they  had  been  allowed  to  vary.  Fac- 
tort  that  fall  in  thia  "gray  area"  may  have  conatrainta  impoaed  upon  them 
in  auch  a  manner  that  the  more  detailed  analyaia  to  be  performed  in  the 
optimisation  proceaa  will  indicate  final  diapoaition.  For  example,  if  a 
queationable  factor  might  have  a  monotonic  influence  on  effectiveneaa,  con- 
aide  ration  of  only  extreme  valuea  might  be  all  that  ia  neceaaary  to  deter¬ 
mine  the  aignificance  of  thia  influence. 

It  ia  important  that  factora  aelection,  variability,  and  the  final  choice 
of  ayatem  definition  be  clearly  indicated  ao  that  the  acope  of  the  optimisa¬ 
tion  proceaa  will  be  known  and  area*  for  poaaible  modification  of  the  formal 
mathematical  aolution  will  be  made  explicit. 

In  many  area*  coat-effectiveneaa  criteria  or  meaaurea  are  commonly 
accepted.  These  are  often  stated  in  terms  of  dollars  per  unit  of  task  per¬ 
formed.  These  measures  are  analogous  to  sales  prices  for  units  of 
measureable  materials  used  in  the  civilian  market,  such  ss  dollars  per 
gallon,  dollars  per  pound,  etc.  These  measures  are  easily  understood  and 
lend  themselves  to  the  spirit  of  the  drive  to  produce  or  purchase  the  most 
for  the  least.  Table  II  lists  some  example*  of  cost  effectiveness  criteria 
and  the  field  of  endeavor  in  which  they  are  used 


TABL’"  n 


EXAMPLES  OF  COST-EFFECTIVENESS  CRITERIA 
IN  VARIOUS  AREAS  OF  ENDEAVOR 


Area  of  Endeavor 


Non-Military: 

Building 

Air  passenger 

Freight 

Computer 

Communications 

Electricity 

Gas 

Public  highways 
Farming 


Military: 

Launch  vehicles 

Satellites 

Missiles 

Interceptors 


Example  of  a 

Cost  Effectiveness  Criterion  * 


Dollars 

Dollars 

Dollars 

Dollars 

Dollars 

Dollars 

Dollars 

Dollars 

Dollars 


per  square  foot 
per  passenger  mile 
per  ton  mile 
per  bit 

per  message  unit 
per  kilowatt  hour 
per  cubic  foot 
per  mile 
per  acre 


Dollar  c  per  pound  payload  in 
orbit 

Dollars  per  hour  of  successful 
operation  in  orbit 
Dollars  per  kill 
Dollars  per  intercept 


*  Cost  per  successful  effort  is  different  for  military  than  for  non-military 
since  success  is  usually  probabilistic  in  nature  in  the  military 
situations. 
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C.  IDENTIFY  AND  SYNTHESIZE  ALTERNATIVES  (TASK  4) 

The  ability  to  optimize  a  system  depends  on  the  availability  of 
alternate  means  of  meeting  the  requirements.  Alternatives  include  the 
means,  approaches,  or  techniques  which  can  be  employed  to  meet  the 
stated  requirements  within  the  constraints  of  the  resources.  Obviously,  if 
no  alternatives  present  themselves  or  if  they  are  ruled  out  by  the  statement 
of  requirements  and  resources,  there  is  no  problem  in  selection.  It  also 
follows  that  when  alternatives  do  present  themselves,  decision  between 
them  is  required.  1 1  the  system  is  to  be  optimized  with  respect  to  cost 
effectiveness,  then  the  optimization  process  must  extend  to  each  decision 
made  on  the  alternatives  presented. 

Table  111  shows  an  example  of  the  types  of  alternatives  considered  in 
optimization  studies. 

It  is  possible  to  arrive  at  the  optimum  system  of  a  given  type  by  de¬ 
signing  a  great  number  of  alternative  systems,  estimating  cost  and  effec¬ 
tiveness  for  each,  and  simply  selecting  the  best  one.  However,  the  large 
number  of  man-hours  required  render  such  an  approach  impractical.  As  a 
result,  it  is  necessary  to  consider  only  a  very  few  basic  configurations  or 
candidate  systems  within  a  given  system  type.  A  completely  adequate  cost 
effectiveness  optimization  of  the  system  can  often  be  accomplished  with  as 
little  as  one  basic  configuration.  However,  due  to  the  small  number  of 
basic  configurations  thus  explored,  it  is  necessary  that  each  basic  configu¬ 
ration  be  optimized  within  itself.  This  is  accomplished  by  synthesizing  and 
evaluating  variations  or  alternatives  at  several  levels  within  the  basic  con¬ 
figuration.  These  alternatives  may  take  the  form  ot  either  physical  or 
performance  characteristics. 

Each  military  system  has  a  number  of  physical  characteristics  that 
affect  cost,  performance  and  effectiveness.  A  list  of  physical  characteris¬ 
tics  to  cover  all  systems  will  not  be  attempted.  A  few  of  those  common  to 
most  systems  include  weight,  volume,  shape,  energy  levels,  mechanical 
and  electrical  packaging,  and  environmental  capabilities.  The  physical 
characteristics  of  a  system  affect  the  cost  elenirnts  incurred  in 


TABLE  m 


TYPICAL  ALTERNATIVES 
POSSIBLE  IN  COST-EFFECTIVENESS 
OPTIMIZATION 

Basic  Concept 

Manned  Versus  Unmanned 
Liquid  Versus  Solid  Rockets 
System  and  Subsystem  Type 

Battery  power  versus  generation 
Materials  Choice 

System  and  subsystem  configuration 
Redundancy 
Maintenance 

Hi>Reliability  versus  MIL  Std  Parts 


Operational  modes 


development,  procurement  and  support.  There  is  obviously  a  broad  range 
of  cost  sensitivity  as  cost  elements  are  compared  for  different  design  alter¬ 
natives  of  a  given  system  requirement  as  well  as  for  different  technology 
alternatives  within  a  given  design  alternative. 

When  one  considers  the  area  of  performance  characteristics  of  military 
system*,  it  is  difficult  to  prepare  a  comprehensive  listing,  and  few  pevfor- 
mance  characteristic e  are  common.  Typical  performance  characteristics 
for  a  few  military  systems  include:  accuracy,  speed,  thrust,  memory 
capability,  computational  capability,  signal  to  noise  ratio,  range,  power 
output,  discrimination,  etc.  Relationships  between  cost  elements  and  per¬ 
formance  characteristics  are  fertile  areas  for  optimisation.  A  particular 
cost  element  will  vary  as  the  performance  characteristic  varies  over  the 
range  of  values  possible  for  the  design  alternative.  For  a  given  require¬ 
ment  level  of  .a  performance  characteristic,  cost  element  variation  as  a 
function  of  the  different  design  alternatives  and  technology  alternatives 
within  a  design  alternative,  are  of  prime  importance.  The  constraints  on 
performance  characteristics  are  generally  set  by  scientific,  engineering, 
and  manufacturing  knowledge  and  capabilities. 

In  listing  the  alternatives,  primary  importance  should  be  given  to  those 
which  have  a  significant  impact  on  cost  or  the  resources  established  in  the 
statement  of  requirements.  It  is  a  value  engineering  maxim  to  dig  where 
experience  and  engineering  judgment  indicate  there  is  likely  to  be  gold.  A 
preliminary  analysis  of  an  initial  system  design  can  indicate  the  major 
impact  areas. 

The  number  of  alternatives  to  be  considered  in  the  optimization  process 
can,  in  many  cases,  be  reduced  by  screening  these  alternatives  against  the 
available  resources  established  in  the  statement  of  requirements.  In  the 
area  of  cost,  physical  characteristic  constraint  relations  established  out¬ 
side  the  cost  area  will  often  bound  and  limit  the  feasibility  or  scope  of 
alternatives. 

As  an  xample  of  such  screening,  let  us  look  at  a  case  wherein  an 
isotope  power  source  is  being  considered  as  an  alternative  to  a  power  system 
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design  more  compatible  with  current  state-of-the-art.  If  the  required  date 
for  system  operational  capability  is  relatively  early  in  time,  the  isotope 
power  source  may  be  automatically  ruled  out  by  lack  of  availability  by  the 
required  date. 

An  example  of  an  alternate  type  of  screening  problem  could  occur  when 
comparing  the  same  isotope  energy  source  against  an  operational  date  stated 
as  a  variable.  Assuming  that  system  effectiveness  or  value  decreases  as 
the  operational  date  is  delayed,  it  may  be  possible  to  eliminate  the  isotope 
energy  source  from  further  detailed  consideration  on  the  basis  that  the  cost 
or  effectiveness  gains  associated  therewith  do  not  compare  favorably  with 
the  value  or  effectiveness  lost  due  to  the  corresponding  slide  in  operational 
date. 

In  preparing  a  list  of  alternatives,  one  should  associate  them  with  the 
l.ivel  at  which  decisions  upon  the  alternatives  are  to  be  made.  At  system 
level,  decisions  should  be  made  on  alternatives  which  impact  on  the  basic 
system  configuration  or  operational  mode.  Decisions  which  do  not  directly 
or  substantially  affect  basic  configuration  and  operational  mode  should  be 
made  at  lower  levels  using  trade-off  factors  developed  for  the  entire  sys¬ 
tem.  If  such  lower  level  decisions  are  attempted  as  a  part  of  the  ftver-all 
system  optimisation  process,  the  scope  of  the  system  level  problem  may 
become  unmanageable.  It  is  recognised,  however,  that  the  basic  system 
may  change  significantly  as  a  result  of  optimisations  at  system  level.  Fur¬ 
ther,  the  trade-offs  and  optimisations  made  at  subcontractors  level  with  a 
single  sub-assembly  or  black  box  may  have  far-reaching  effect  on  system 
effectiveness.  Thus,  any  system  for  handling  cost  effectiveness  must  per¬ 
mit  optimisation  to  feed  both  up  and  down  through  the  various  system  levels 
and/or  tiers  of  customer /cont ractor /subcontractors.  The  process  of 
feeding  up  and  down  through  the  system  must  be  recognised  as  an  iterative 
one, wherein  it  may  be  necessary  to  reiterate  some  of  the  lower  level  sub- 
of  timisatiens  to  insure  that  the  basic  system  changes  have  not  altered  pre¬ 
viously  established  conclusions. 


D.  IDENTIFY  VARIABLES  {TASK  5) 

For  each  alternative  on  which  a  decision  must  be  reached,  one  can 
proceed  to  list  the  significant  variables  which  should  be  considered  in  the 
optimization  analysis.  Table  IV  shows  examples  of  variables  which  can 
influence  the  choice  of  alternatives.  In  a  cost-effectiveness  optimization, 
it  is  evident  that  although  many  variables  exist  and  could  influence  final 
selection  of  an  alternative  approach,  the  variables  which  are  significant 
can  be  limited  to  those  which  have  an  impact  on  cost,  resouices  available, 
or  the  effectiveness  with  which  the  system  performs  its  function.  Variables 
which  do  rot  influence  these  quantities  significantly  should  not  be  included 
in  the  optimization  process. 

Variables  can  also  be  screened  to  a  certain  extent.  In  general,  some 
variables  can  be  arbitrarily  treated  as  fixed  quantities  as  a  result  of  ele¬ 
ments  in  the  statement  of  requirements,  limitations  on  resources,  or  other 
previously  established  decisions  on  the  program.  In  other  cases,  a  legiti¬ 
mate  variable  can  he  treated  as  a  fixed  quantity  initially.  Then,  after  initial 
optimizations  have  been  completed,  the  effect  of  altering  the  variable  can  be 
expressed  in  terms  of  impact  on  the  final  answer.  In  many  cases,  judicious 
fixation  of  variables  in  this  manner  can  save  a  large  amount  of  manpower 
expenditure  if  the  decision  to  fix  the  variable  is  based  upon  probable  insen¬ 
sitivities  of  the  answer  to  the  magnitude  of  variation  expected. 

The  range  of  each  variable  to  be  considered  should,  for  economy  of 
analysis  effort,  be  limited.  Constraints  on  physical  characteristics  often 
limit  the  range  of  performaice  characteristics  or  other  variables  which  can 
be  considered.  Preliminary  sensitivity  analysis,  rough-cut  analysis,  or  an 
extreme  (maximum  and  minimum)  value  analysis  are  also  useful  in  indicat¬ 
ing  probable  limits  of  variables.  Variables  thus  limited  should  be  re¬ 
examined  after  completion  of  the  optimization  study.  If  a  definite  optimum 
point  is  reached  within  the  limits  of  each  variable,  it  is  generally  safe  to 
assume  that  the  limits  established  were  reasonable. 


TABLE  IV 


TYPICAL  VARIABLES  INFLUENCING 
COST-EFFECTIVENESS  EVALUATION  OF  ALTERNATIVES 

Cost 

Weight 

Payload  Carried 
Mission  Length 
State -of -Art 
Time  Required 
Reliability 
Safety 

Maintenance 

Availability 

Vulnerability 

Survivability 


SECTION  IV 


COST  -EFFECTIVENESS 

MODELS,  CONCEPTS,  VALIDATION  AND  SENSITIVITY 

(TASKS  6  fc  7) 

A.  MATHEMATICAL  FORMULATION  OF  EFFECTIVENESS  AND  COSTS 

A  bi. sic  problem  which  confronts  any  military  planner  is  that  of  pro¬ 
viding  assurance  that  he  is  "buying  the  most  for  the  militarv  dollar.  "  In 
general  terms,  the  military  service  procures  a  system  (or  improvement 
thereon)  with  effectiveness  or  potential  effectiveness  E  (or  effectiveness 
improvement  AE).  The  system  is  paid  for  in  resources  which  are  usually 
related  to  dollar  costs  C  (or  AC  in  case  of  the  improvement).  The  military 
planner  wants  to  be  sure  that  units  with  the  highest  E  are  obtained  for  the 
C  units  expended. 

1.  Effectiveness  as  a  Function  cf  Various  Parameters  The 
effectivenecs  (E)  of  any  system  is,  in  part,  a  function  of  parameters  (X^) 
which  are  controllable.  Some  of  these  parameters  may  be  constrained  by 
the  state-of-the-art  (Xi^)  at  a  particular  point  in  time.  Unfortunately.  E 
is  also,  in  part,  a  function  of  factors  not  under  the  control  of  designer, 
purchaser,  or  user.  These  include  characteristics  of  weapons  in  the 
enemy  arsenal,  or  of  targets  for  the  subject  weapon  system.  Thus,  E 
will  be  degraded  with  time  (t)  if  the  enemy  capability  improves.  (This 
degradation  function  may  be  discontinous.  For  example,  E  may  be  con¬ 
stant  and  high  until  some  point  in  time  when  the  enemy  has  developed  and 
deployed  a  counter  weapon.  At  this  point  E  may  suddenly  drop  to  or  near 
zero.  )  E  is  also  a  function  of  time  when  one  is  either  procuring  a  new 
system  or  improving  an  existing  system.  Thus,  the  effectiveness  for  a 
new  system  is  ordinarily  zero  until  the  development  is  completed  and  the 
deployment  of  the  operational  system  is  initiated.  —  Chang*  of  effectiveness 


» / 

—  Under  concurrency,  some  limited  capability  may  exist  in  the  so-called 
IOC  (Initial  Operational  Capability)  phase,  when  "Development,  "  in  a 
formal  sense,  may  be  far  from  complete. 


of  an  existing  system  does  not  occur  until  modification  is  initiated.  We  may 
write 

E  =  E(X.,t),  0<X.<XL  (1{ 

i 

where  all  design  parameters  have  been  defined  so  that  they  are  positive  and 
limited  it  some  upper  bound. 

Figure  4,  which  displays  a  sample  effectiveness  function  plotted 
against  time,  is  intended  to  illustrate  some  of  the  factors  which  contribute 
to  effectiveness.  Projections  of  effectiveness  must  be  updated  frequently 
so  that  sufficient  lead  time  is  allowed  for  procuring  new  or  modifying  ex¬ 
isting  systems  to  replace  those  whose  effectiveness  is  either  declining  or 
inadequate. 

Figure  5  shows  effectiveness  as  a  function  of  two  controllable  (i.  e., 
independent  variable)  parameters.  For  illustrative  purposes,  mean  time  to 
failure  and  the  reciprocal  of  mean  down  time  per  failure  (which  can  be  called 
the  repair  rate)  were  used  as  variables.  Limits  of  »he  state-of-the-art  are 
shown  as  functional  limits  rather  than  as  strictly  constant  limits. 

The  controllable  variables  which  are  important  depend  upon  the  type 
of  system  and  the  stage  in  planning,  development,  or  employment  being 
considered.  For  example,  in  planning  an  atmospheric  bombing  system, 
force  size,  speed,  range,  payload,  altitude,  availability,  reaction  time, 
turn-around  capability,  invulnerability,  penetrability,  accuracy,  etc.  would 
all  be  of  interest. 

During  the  development  and  procurement  stages,  availability  may  be 
of  paramount  importance,  and  may  generate  a  high  level  of  interest  in  both 
failure  rates  and  repair  rates.  Oth<-r  factors  such  as  speed,  range,  payload, 
etc.  could  reenter  the  picture  if  either  the  planning  estimate  for  the  state-of- 
the-art  or  the  prediction  of  cost  were  found  to  be  in  error  at  a  later  time. 

For  a  defensive  radar  system,  these  variables  could  become  the  number  of 
radars,  seaiiii  iange,  lock-on  range,  target  capacity  per  unit,  angular 
accuracy,  range  accuracy,  availability,  false  alarm  rates,  etc.  . 

The  appropriate  unit  of  measurement  for  effectiveness,  in  general, 


EFFECTIVENESS 


FIGURE  5 


AVAILABILITY  AS  A  FUNCTION  OF  TWO  CONTROLLABLE 
PARAMETERS:  REPAIR  RATE  AND  MEAN  TIME 
TO  FAILURE 


depends  upon  what  mission  the  system  is  expected  to  perform.  Thus,  a 
bombing  or  missile  system's  effectiveness  could  be  related  to  the  number  of 
enemy  targets  of  a  given  kind  which  could  be  destroyed  by  an  assumed  force 
in  a  prescribed  time  under  prescribed  conditions.  A  defensive  radar  system 
could  be  rated  according  to  the  number  of  targets  an  assumed  net  could 
handle  under  given  conditions.  However,  it  is  possible  to  formulate  effec¬ 
tiveness,  £,  so  that  all  systems  may  be  related  to  a  common  measure.  One 
alternative  is  the  expected  fraction  of  the  total  number  of  required  individual 
missions  within  a  class  which  would  be  completed  successfully. 

2.  Cost  as  a  Function  of  Various  Parameters  The  cost  (C)  for 
obtaining  a  system  with  effectiveness  (E)  is  also  a  function  of  the  param¬ 
eters  X^.  Like  E,  C  also  depends  on  time,  but  in  a  different  manner.  C  is 
influenced  by  the  speed  with  which  one  may  want  to  obtain  a  system  with  a 
given  level  of  effectiveness.  Urgency  m.iy  lead  to  parallel  developments  on 
critical  items  involving  risk  or  uncertainty,  and  certainly  will  lead  to  over¬ 
time  and  waste  due  to  errors. 

Estimates  of  time  to  completion  (for  a  given  level  of  project  effort) 
can  be  obtained  from  PERT  networks.  Combination  of  this  with  PERT-Cost 
analyses  or  the  use  of  other  techniques  (e.  g..  Critical  Path  Scheduling)  can 
lead  to  plots  of  cost  versus  time.  A  conceptual  trade-off  is  shown  in 
Figure  6,  with  effectiveness,  E, as  a  third  parameter. 

The  curves  of  Figure  6  are  asymptotic  to  lines  parallel  to  both  the 
cost  and  time  axes.  However,  past  experience  has  shown  that  adding  more 
dollars,  when  near  minimum  development  time,  may  actually  cause  time 
delays.  Extension  ol'  project  "deadlines,"  as  a  revision  of  an  original  well- 
planned  program  not  only  can,  but  usually  does,  lead  to  increased  cost. 

From  the  foregoing,  it  should  be  evident  thai  the  cost  of  obtaining  a 
new  system  or  improving  an  existing  one  is  a  function  of  i  ont Tollable 
system  parameters  (X^)  and  development  time  (t^).  Thus, 


Useful  life  and,  therefore,  "long-term  effectiveness"  of  the  system 
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FIGURE  fe 

DEVELOPMENT  COST  VERSUS  DEVELOPMENT  TIME 
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FIGURE  6 


DEVELOPMENT  COST  VERSUS  DEVELOPMENT  TIME 

(£,<e2<  e3) 
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only  be  accurately  calculated  after  the  fact  and  often  haa  low  accuracy  if 
estimated  at  any  point  iu  the  life  of  the  system.  During  development  and 
procurement,  both  modification  costs  and  the  effect  of  modifications  on  such 
things  as  system  effectiveness  and  operational  and  support  costs  cannot  he 
accurately  assessed.  If  they  could,  such  modifications  would  probably  have 
been  included  in  the  initial  design.  Furthermore,  at  any  point  in  the  system 
lifetime,  only  those  costs  still  to  be  incurred  are  of  interest;  those  already 
made  are  unrecoverable. 

The  foregoing  discussion  of  effectiveness,  cost,  and  time  was  in¬ 
cluded  here  to  provide  a  meaningful  background  for  the  consideration  of  the 
cost-effectiveness  models  which  follow.  Detailed  consideration  of  these 
factors  (i.  e.,  effectiveness,  cost,  and  time)  and  models  for  their  determina¬ 
tion  are  given  in  other  portions  of  this  report.  It  is  sufficient  to  note  here 
that,  where  C  is  total  cost  and  is  the  i***  cost  element,  C  is  given  by 


Often  effectiveness  will  be  measured  in  terms  of  a  probability  of  success. 

If  E  is  effectiveness,  and  P(  is  the  ith  conditional  probability  element,  E  is 
given  by 


E  •  «  Pj  (6) 

*  Pavall.  ^launch /avail,  ^flight /launch 
‘  Ppenetr ate /flight’  ’  ’  *tc’ 

The  problem  is  to  define  and  evaluate  cost-effectiveness  models  which 
can  be  used  to  choose  an  "optimum”  configuration  for: 

(1)  a  new  system,  or 

(2)  modification*  to  an  existing  system. 


three  measures  which  depend  upOu  system  parameter  choices  are  used 

here  to  determine  when  a  beat  configuration  has  been  achieved.  These  are: 


(1)  cost  (C); 

(2)  effectiveness  level  (E); 

(3)  long-term  effectiveness  (Ig)  which  involves:  (E);  time  of 

activation  --  related  to  development  time  (t^);  and  time  of 

obsolescence  (t  ). 

o 

It  is  fairly  certain  that  the  optimization  or  decision  process  will 
occur  under  various  constraints.  These  may  include: 

(1)  Total  cost  (C)  is  either  given  or  constrained  by  an 
upper  bound; 

(2)  effectiveness  (E)  is  specified  or  constrained  by  a 
lower,  bound; 

(3)  time  of  activation  (t^)  and  cost  (C)  are  specified 
or  constrained  by  an  upper  bound; 

(4)  time  of  activation  (t^)  and  effectiveness  (E)  are 
specified  c?  constrained  by  upper  and  lower  bounds 
respectively; 

(3)  some  system  performance  parameters  (example: 
missile  range)  may  be  specified  or  bounded. 

All  of  the  dependent  factors  are  functions  of  controllable 
parameters  (ignoring  the  problems  of  risk  and  uncertainty).  Thus, 

C  *  C  (X..  td) 

E  *  E  (X.) 

V  \,<Xi> 

lE  =  (to  *  td)  E 

0  <  X.  <  X. 

—  i  -  L. 

i 

where  t^  has  been  selected  as  a  controllable  variable.  For  the  present 


(7) 


discussion,  we  will  assume  the  E  is  constant  during  the  useful  system  life, 
and  zero  elsewhere. 

Although  the  above  equations  have  been  written  so  that  symbols  are 
obviously  these  for  development  •procurement,  what  follows  is  just  as 
applicable  for  system  modification. 

It  is  a  natural  human  trait  to  seek  conciseness  in  considering  measures 
of  value  (e.g. ,  Cost- Effectiveness).  Thus,  we  are  seeking  cost-effective¬ 
ness  models  which  provide  measures  in  terms  of  cost  units/mission  or 
cost  units /unit  of  time/missioo.  Examples  include  dollar/kill  of  100  psi 
point  target,  dollar /year /ICBhi  detected,  etc. 

C.  SOME  POSSIBLE  MODELS 

In  the  following,  only  three  basic  cost-effectiveness  models  arc  con¬ 
sidered.  This  should  not  be  interpreted  as  indicative  that  other  models  are 
of  no  value.  Indeed,  the  reader  is  encouraged  to  broaden  his  study  by 
looking  elsewhere.  However,  the  three  models  discussed  here  should  be 
suffici'-nt  to  exemplify  the  factors  of  importance. 

The  models  considered  here  are  what  may  be  referred  to  as  the  "profit," 
the  "cost-effectiveness  (level)  ratio,"  and  the  "cost-effectiveness  (long 
term)  ratio"  models.  Variations  on  these,  usually  in  terms  of  constraints, 
are  also  discussed  but  are  not  considered  as  constituting  a  separate  basic 
model  type. 

1.  Profit  Model  The  profit  model  is  simply  the  application  of  the 
commercial  concept  of  return  on  investment.  This  may  be  stated  ir  terms 
of  absolute  return: 

P  «  E  -  C 

■  Value  Received  -  Cost  Expended  (8) 

or: 

•  Value  expected  -  Cost  expected. 

or  rate  of  return 

r  ■  •  (9) 

The  usefulness  of  these  models  is  contingent  on  solution  of  the  rather 
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difficult  problem  of  finding  a  common  unit  of  measure  for  E  and  C.  This 
has  been  done  in  the  past  by  such  arbitrary  means  as  indexing  each  on  a 
common  scale  (e.  g.,  0  to  100)  or  by  relating  E  to  value  of  targets  killed, 
value  of  property  defended  or  protected,  etc. .  However,  such  arbitrary 
actions  --  whatever  the  logic  upon  which  they  are  based  --  often  lead  to 
pros*  misunderstandings  and  frustrations  or.  the  part  of  those  involved  in  the 
decision-making  process.  As  an  illustration,  consider  the  case  of  an  air- 
to-air  missile  designed  to  be  used  against  bombers  which  are  attacking 
important  targets.  Let  cost,  as  a  function  of  in-flight  reliability,  R,  be 
given  by 


C  =  10000  ♦ 


1000 


(10) 


and  let  effectiveness,  in  terms  of  single  shot  kill  probability,  be  given  by 


E  =  .  80R  . 


(U) 


Now  let  the  value  of  effectiveness  (Vj,)  be  given  by 


Ve«KE  (12) 

where  K  is  an  arbitrary  constant.  Some  may  wish  to  determine  K  from  the 
value  of  the  bomber  (there  is  a  possible  wide  variety  of  bomber  values),  or 
values  of  the  target  which  the  bomber  might  attack  (an  even  wider  variety), 
or  both.  Mathematical  manipulations  lead  to  the  conclusion  that  the  optimum 
reliability  is  given  by 


R  *  1  -  (HPR)  for  K  >  1250.  (13) 

Extremely  high  values  for  K  can  lead  to  an  R  precariously  near  1.  When 
this  is  substituted  back  into  the  cost  formula,  a  rather  costly  missile  may 
result.  Admittedly,  the  foregoing  example  suffers  from  oversimplification. 
It  is  a  suboptimisation  in  an  extremely  complex  problem.  However,  it  is 
indicative  of  the  importance  of  the  problem  of  selecting  a  value  scaling 
factor  (or  function)  for  the  "profit"  model. 

The  second  "profit"  model  (i.e.,  rate  of  return)  suffers  from  an 


additional  difficulty  occurring  with  ratio  functions.  Under  some  circum¬ 
stances,  the  optimum  may  occur  at  t  *0  origin  and  one  finds  that  "the  best 
system  version  is  no  system  at  all.  "  As  an  example  of  this,  consider  the 
cost  and  effectiveness  functions: 

C  =  AX  (14) 

where  X  is  some  system  parameter  related  to  effectiveness  by 

E«B(l-e’DX)  (15) 

and  A  and  B  are  constants  of  proportionality. 

Substitution  of  (14)  and  (15)  into  (9)  leads  to  an  optimum  with  X  *  0.  On  the 
other  hand,  switching  the  C  and  E  functions  leads  to  X  *  •  for  the  optimum. 
Thus,  in  the  first  case,  the  maximum  rate  of  return  occurs  when  we  spend 
no  resources  or  only  a  minute  amount.  In  the  second,  we  should  spend  all 
the  resources  at  our  disposal.  However,  let  us  return  to  reality.  The  first 
system  (optimum  at  X  ■  0)  may  still  be  an  extremely  good  system  at  non- 
optimum  points  when  compared  with  other  systems  available  for  the  same 
mission.  Further,  it  must  be  remembered  that  we  may  be  considering  a 
useful  mission  which  we  must  be  prepared  to  perform.  Thus,  the  negative 
conclusion  may  be  either  incorrectly  or  incompletely  stated. 

No  extensive  discussion  of  solutions  to  this  dilemma  will  be  presented 
here.  However,  if  one  is  enamored  with  rate  of  return  as  a  concise  way  of 
presenting  the  value  of  a  system,  then  one  should  remember  that,  in  general, 
side  conditions  (constraints)  are  usually  involved  in  this  evaluation  process. 
Thus,  for  example,  we  may  be  interested  only  in  systems  for  which  effec¬ 
tiveness  is  above  a  certain  level  and  we  have  an  upper  limit  on  resources 
available.  We  may  then  maximise  r  with  equality  constraints  for  varying 
levels  of  either  or  until  the  other  constraint  is  encountered.  Thus, 
stated  mathematically, 


Maximise  !  r  • 

L. 


ElXj)  -  CfXj)  - 

— cnri — J 


(16) 


MX.)>EC 

(17) 

c<X|><cc 

(18) 

to 

with  constraints 


The  results,  in  term*  of  r,  may  then  be  plotted  versus  C  and/or  E  and 
decisions  made  on  some  rational  basis.  It  should  be  noted  that  this  is 
equivalent  to  the  following: 

Maximise  [EfX^)]  with  constraint  QX^)  -  C^.  =  0  (19) 

or 

Minimise  [CfX^]  with  constraints  ECXj)  -  Ec  =  0  (20) 

at  each  of  several  constraint  values  (E^.  or  C^.  as  appropriate)  and  then 
examining  the  results  in  terms  of  r. 

If  some  single  function  for  the  value  of  E  can  be  found  to  represent  the 
rational  process  followed  in  the  final  decision,  then  an  optimum  may  be 
obtained  in  a  direct  manner.  However,  many  factors  involved  in  decision* 
making  are  either  not  reducible  to  a  mathematical  expression,  or  can  only 
be  represented  in  an  imprecise  way.  The  model  can  still  provide  a  useful 
tool  in  reducing  the  scope  of  the  area  which  must  ultimately  be  investigated 
in  a  more  rational  way. 

The  other  dilemma  (optimum  at  X  ■+  *)  will  not  be  discussed  further 
here  since  it  is  analogous  to  the  first  problem. 

Consideration  of  Ig  (i.  e.,  long  range  or  integrated  effectiveness)  in 
either  version  of  the  profit  model  is  different  from  conoid* ration  of  E,  itself 
by  virtue  of  lifetime  support  costs  in  large  part.  The  element  of  time  is 
introduced  along  with  significant  costs  of  support.  Since  the  implications 
of  time  are  more  readily  seen  in  terms  of  the  cost-effectiveness  (long  range) 
ratio  model,  discussions  of  this  measurement  concept  will  be  given  later. 

2.  Cost-Effectiveness  (Level)  Ratio  Model  As  indicated  by  the  name, 
this  model  is  given  by 

=el  *  % 

The  model  has  the  advantages  of  providing  measure  values  in  natural  terms. 
Thus,  in  terms  analogous  to  transportation  (cents  per  ton-mile  or  cents 
per  passenger-mile),  weapon  cost-effectiveness 
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values  may  be  given  in  terms  of  dollars  per  kill,  dollars  per  intercept,  or 
other  measures  of  dollars  per  mission.  However,  we  are  again  faced  with 
the  dilemma  associated  with  ratio  functions.  This  problem  has  already 
been  considered  in  relation  to  the  rate  of  return  version  of  the  Profit  Model. 
As  in  that  case,  a  useful  approach  is  that  of  employing  Lagrange  multipliers 
(see  references  4  and  5)  to  find  extrema.  Such  a  problem  might  be  stated  as 

Maximise  E  +  XC(C  Cc)  +  Xt(td  -  tc),  (22) 

where  X^.  and  X(  are  the  Lagrangian  multipliers  for  the  cost  and  time  con¬ 
straints  respectively,  or 

Minimise  C  +  Xg(E  -  Ec)  +  Xt(td  -  tc).  (23) 


The  cost-effectiveness  value  of  the  system  may  then  be  presented  in 
terms  of  Cgg  (as  computed  from  Equation  (21))  and  plotted  vorsus  C(.  or 
E^.  as  appropriate.  For  simplicity  of  presentation,  constraints  have  been 
indicated  in  (22)  and  (23)  through  the  use  of  undetermined  Lagrange  multi¬ 
pliers  without  regard  to  any  mathematical  questions  which  may  arise.  A 
time -of -activation  constraint  has  also  been  shown  to  indicate  how  this 
factor  may  be  introduced  into  this  type  of  model  for  cost-effectiveness 
considerations.  Other  constraints  may  be  encountered,  such  as  those  im¬ 
posed  on  system  design  parameters  by  either  state-of-the-art  or  other 
causes. 

3.  Long  Range  Cost-Effectiveness  Ratio  Model  The  functional 
relationship  for  this  model  is  given  by: 

CELR  '  <24> 

Considering  the  assumed  relationships  given  in  Section  IV-B.  Statement  of 
the  Optimisation  Problem,  the  equation  may  be  transposed  to: 


(25) 


This  form  displays  the  sometimes  desired  measures  of  value  in  terms  of  dol¬ 
lar/yea  Emission.  This  combines  the  relevant  factors  of  cost,  longevity  and 
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effectiveness  potential  into  a  single  measure. 

D.  OPTIMIZATION  TECHNIQUES 

The  use  of  the  technique  of  undetermined  (Lagrangian)  multipliers  as  a 
tool  of  optimisation  was  mentioned  above.  This  technique  is  useful  when 
well  defined  analytical  relationships  exist  among  the  variables,  and  when 
the  constraints  are  expressly  stated  as  fixed,  single  valued  requirements. 
Alternative  techniques  are  preferable  when  the  relations  among  the  variables 
are  empirical,  disjointed,  or  discrete  and  whenever  the  constraints  are 
stated  as  a  range  of  acceptable  values.  For  example,  when  a  finite  number 
of  discrete  alternatives  exist,  optimisation  would  ordinarily  be  accomplished 
by  the  straight-forward  procedure  of  direct  comparison  of  the  calculated 
cost  and  predicted  effectiveness  of  each  alternative. 

When  the  data  is  empirical,  as  opposed  to  analytical,  graphical  tech¬ 
niques  will  usually  prove  to  be  more  useful  and  are  particularly  useful  when 
the  constraints  are  given  as  a  bounding  range  of  acceptable  values. 

In  addition  to  the  above  techniques,  there  are  a  number  of  others 
discussed  in  the  literature.  Among  the  more  common  are: 

•  marginal  analysis 

•  dynamic  programming 

•  simple  maxi:ni  nation 
Pontryagin's  maximum  principle 

•  linear  programming 

•  calculus  of  variations 

•  method  of  steepest  ascent 

•  ''mini-max  principle1'  of  the 
theory  01  games. 

In  a  report  of  this  length  it  would  be  impossible  to  present  an  intelligible 
illustration  of  each  of  these  techniques.  The  task  group  has  therefore 
arbitrarily  chosen  to  limit  its  examples  to  illustrations  of  those  methods 
which  are  simplp  to  grasp,  easy  to  exploit,  t  .d  have  a  fairly  wide  applica¬ 
tion  to  reality,  namely:  exhaustion  of  feasible  alternatives;  graphical  tech¬ 
niques;  simple  maximisation;  ar.d  dynamic  p -op ramming. 
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This  limitation  should  not  be  construed  as  an  unqualified  endorsement  of 
these  techniques  to  the  exclusion  of  the  others. 

Further  discussion  of  optimisation  including  principles,  criteria  and 
techniques  is  contained  in  Section  II  of  Volume  III  of  the  final  report  of 
Task  Group  IV. 

E.  MODEL  VALIDITY  AND  SENSITIVITY. 

1.  Assumptions  All  assumptions  required  for  the  model  should  be 
explicitly  stated  and,  if  possible,  supported  by  factual  evidence.  If  no  such 
evidence  exists,  it  is  advisable  to  state  the  reason  for  the  assumption  (like 
mathematical  simplicity)  in  order  to  indicate  the  degree  to  which  the 
assumptions  will  require  further  justification,  and  to  pinpoint  the  areas  in 
which  errors  might  be  introduced. 

2.  Adequacy  A  model  must  be  adequate  in  the  sense  that  all  major 
variables  to  which  the  solution  is  sensitive  are  quantitatively  considered, 
where  possible.  Many  of  these  variables  will  have  been  preselected. 
Through  manipulation  of  the  model,  some  of  the  variables  may  be  excluded 
or  restricted,  and  others  may  be  introduced.  Non-quantifiable  variables 
must  be  accounted  for  by  modification  of  the  solution  rather  than  by  direct 
incorporation  into  th*  model.  In  this  sense  they  become  quantifiable. 

3.  Representativeness  Although  no  model  can  completely  duplicate 
the  "real  world,  "  it  is  required  that  the  model  reasonably  represent  the 
true  situation.  For  complex  problems,  this  may  be  possible  only  for  sub¬ 
parts  of  the  problem,  which  must  be  pieced  together  through  appropriate 
modeling  techniques.  As  an  example,  analytic  representation  may  be  pos¬ 
sible  for  various  phases  of  a  complex  maintenance  activity.  The  outputs 
from  these  aitaiyses  may  then  be  used  as  inputs  to  a  simulation  procedure 
for  modeling  the  complete  maintenance  process. 

4.  Probabilistic  Aspects --Risk  and  Uncertainty  The  various  types 
of  unknowns  involved  in  the  problem  cannot  be  ignored,  nor  can  they  be 
"assumed"  out;  they  must  be  faced  squarely.  There  may  be  technological 
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uncertainties  involved  with  some  of  the  system  alternatives,  operational 
uncertainties  involved  with  planning  and  carrying  out  the  mission,  uncer¬ 
tainties  about  enemy  strategy  and  action,  and  statistical  uncertainties 
governed  by  the  laws  of  chance  (referred  to  as  risk).  Risk  analysis  may  be 
applied  in  cases  involving  statistical  uncertainties,  func  t  ions- yf- random - 
variables  theory  or  such  procedures  as  Monte  Carlo  techniques.  The  non- 
quantifiable  area  of  uncertainties  is  a  matter  of  expert  judgment.  The  pro¬ 
bability  of  guessing  correctly  for  every  uncertainty  is  quite  small.  The 
general  approach  is  to  examine  all  major  contingencies  and  compute  resul¬ 
tant  cost-effectiveness  parameters.  The  optimisation  criterion,  then, 
must  be  adaptable  for  use  in  the  evaluation  of  the  set  of  cost-effectiveness 
results.  The  developments  of  decision  theory  and  game  theory  conceptually 
become  most  applicable  in  the  selection  of  a  decision  model  in  these  cases, 
since  different  alternatives  may  be  best  for  different  contingencies. 

5.  Validity  The  final  test  of  the  model  is  whether  or  not  it  yields 
the  best  system.  Unfortunately,  this  determination  can  be  made  only  after 
systems  are  developed  and  in  use,  if  it  can  be  made  at  all.  However,  cer¬ 
tain  questions  will  disclose  weaknesses  that  can  be  corrected: 


(a)  Consistency 

(b)  Sensitivity 

(c)  Plausibility 

(d)  Criticality 

(e)  Workability 

( f)  Suitability 


are  results  consistent  when  major  parameters 
are  varied,  especially  to  extremes  ? 
do  input -variable  changes  result  in  output 
changes  that  are  consistent  with  expectations? 
are  results  plausible  for  special  cases  where 
prior  information  exists? 
do  minor  changes  in  assumptions  result  in 
major  changes  in  the  results? 
docs  the  model  require  inputs  or  computa¬ 
tional  capabilities  that  are  not  available 
within  the  bounds  of  current  technology  ? 
is  the  model  consistent  with  the  objectives; 
i.  e.  .  will  it  answer  the  right  questions0 


F.  CONCLUSIONS 


From  the  foregoiug  discussion  of  basic  model  types,  it  should  be 
apparent  that  no  single  one  of  the  basic  types  will  be  the  best  for  application 
to  all  systems  and  optimisation  problems.  The  selection  of  basic  model 
types  should  be  based  on  consideration  of: 

*  definition  of  mission 

*  form  of  criteria  and  measures 

*  nature  of  constraints 

*  type  and  extent  of  information  furnished 
from  or  deriveable  from  results  of  pro¬ 
ceeding  higher  level  studies. 

Application  of  the  p rofit  models  will  potentially  yield  more  information 
from  the  optimisation  analysis  and  provide  a  better  insight  into  the  econom¬ 
ics  of  the  system  as  they  relate  to  competing  systems  and  missions.  How¬ 
ever,  use  of  the  profit  models  requires  a  great  deal  more  input  data  and 
relationships.  The  problem  of  scaling  effectiveness  and/or  costs  to  equiva¬ 
lent  units  will  generally  be  controversial  and,  in  some  cases,  will  be  im¬ 
possible. 

A  less  difficult  method  for  presenting  a  combined  measure  of  value  for 
a  system  is  the  cost-effectiveness  ratio  model.  The  second  version  (i.  e. , 
long-range  effectiveness)  provides  the  best  vehicle  for  combining  all  factors 
involved  during  the  procurement  process,  since  it  provides  an  indication  of 
the  effects  of  the  important  factors:  cost,  time  to  activation,  longevity,  and 
effectiveness  potential. 

For  the  purpose  of  studying  operational  and  support  procedures  after 
system  actuation- -when  procurement  costs  have  already  been  paid--the 
cost-effectiveness  (level)  ratio  model  ts  probably  most  appropriate  with 
cost  being  given  in  terms  of  $/year  for  operation  and  support. 

If  system  modifications  are  being  considered--after  activation--  the 
cost-effectiveness  (long  range)  ratio  model  again  takes  precedence  with 
(t q  -  t^)  indicating  time  remaining  after  completion  of  modification.  Costs 
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(C)  ai*  the  total  costs,  including  development,  investment  or  indirect  costs 
relevant  to  modification,  and  subsequent  direct  operational  and  support 
costs.  Usually,  modifications  should  be  evaluated  in  terms  of  incremental 
costs,  changes  in  effectiveness  potential,  etc. 

In  any  application  of  ratio  models,  the  real  life  situation  (involving 
budget  factors  and  the  "facts  of  life"  related  to  requirements  for  effective 
systems  plus  problems  associated  with  defining  ratio  functions  so  that  use¬ 
ful  and  meaningful  extrema  may  be  derived)  dictates  that  the  optimisation 
process  be  performed  on  a  basis  of  maximising  effectiveness  (of  constant 
cost)  or  minimising  costs  (of  constant  effectiveness)  with  appropriate  con¬ 
straints  imposed.  Occasionally,  it  may  be  desirable  to  minimise  develop¬ 
ment  time  or  maximise  system  life  with  appropriate  constraints. 

The  formulation  of  models  is  not  the  principal  problem  associated 
with  system  evaluation.  Models  are  somewhat  arbitrary  methods  for  de¬ 
riving  a  measure  of  system  value.  The  choice  of  models  must  ultimately 
be  based  upon  consideration  of  the  meaning  associated  with  the  measure  of 
cost-effectiveness  resulting  from  a  particular  form  of  model.  Caution  must 
always  be  exercised  in  interpreting  model  results. 

The  principal  problem  associated  with  system  evaluation  is  that  of 
developing  methods  for  obtaining  meaningful  estimates  of  values  for  the 
factors  of  cost,  effectiveness  potential,  and  time.  Associated  with  this  is 
the  problem  of  obtaining  valid  and  compatible  input  data  for  use  in  the 
various  models. 

In  the  preceding,  costs  have,  in  general, been  used  in  the  context  of 
economic  units  such  as  dollars.  It  often  becomes  necessary  to  consider 
limited  resources  such  as  skilled  manpower  or  critical  material.  Due  to 
the  characteristics  of  these  limitations,  it  is  sometimes  difficult  to  relate 
such  expen  litures  to  the  dollar  units  in  which  other  cost  factors  are  ordinar¬ 
ily  expressed.  This  difficulty  may  be  surmounted  either  by  considering  the 
problem  as  a  suboptimisation- -if  this  can  be  done  in  a  meaningful  wav--or 
by  introducing  an  appropriate  constraint  (for  the  limited  resource)  into  the 
over-all  optimisation  probl  in. 


46 


SECTION  V 


DEVELOPMENT  OF  COST  INPUTS 
TO  COST-EFFECTIVENESS  MODELS 
(TASK  8.  1) 


A.  OBJECTIVES 

In  the  preceding  chapter,  the  reader  has  seen  several  possible  cost- 
effectiveness  models.  In  each  model  it  was  essential  to  develop  the  costs 
of  each  element  of  major  subsystems  and  weapon  systems  in  order  to 
arrive  at  cost-effectiveness  values.  The  objective  of  this  section  is  to 
outline  factors  of  a  method  for  better  estimating  costs.  The  method  out¬ 
lined  will  enable  one  to  insure  that  all  factors  vital  to  cost-effectiveness 
trade-offs  are  included  in  the  analysis. 

We  refer  to  cost  estimates  that  are  done  as  part  of  the  procedures  for 
Air  Force  selection  of  optimum  weapon  systems  at  the  Specific  Operational 
Requirement  (SOR)  and  pre-Request -for-Proposal  (pre-RFP)  stages  of 
weapon  system  procurement.  The  cost  estimates  considered  here  are 
also  the  type  of  parametric  cost  estimates  which  industry  needs  to  help 
select  the  most  cost-effective  designs  for  representing  their  weapon  sys¬ 
tems  approaches  to  the  Department  of  Defense  (DOD).  We  are  not  speaking 
of  the  inherently  proprietary  proposal  cost  data  submitted  at  the  final  stages 
of  weapon  system  procurement.  This  point  must  be  emphasized:  The  costs 
dealt  with  herein  are  to  be  used  for  Air  Force  decision-making  as  to  pre¬ 
ferred  systems  and  are  not  the  costs  in  the  contractors'  proposals  to  build 
vehicles  and  to  make  internal  company  decisions. 

Although  some  method  of  arriving  at  costs  is  mandatory,  the  particular 
approach  presented  here  is  not  to  be  construed  as  "official"  or  "the  only 
acceptable"  approach.  There  should  be  sufficient  flexibility  to  adjust  to 
varying  situations.  It  is  more  important  to  have  a  consistent  definition  of 
cost  categories  and  estimating  procedures  (not  overlooking  important  cost 
factors)  than  to  try  to  insist  on  trie  best  method.  In  the  past,  otherwise 
excellent  weapon  system  studies  have  overlooked  important  cost  categories 


such  as  R  &  D,  trainin  H,  etc.  ,  thus  making  the  subsequent  weapon 
system  comparisons  ur  r  suspect,  to  say  the  least.  Adequate  weapon 

selection  among  alternatives  requires  that  contractor  teams  be  able  to 
develop  cost  breakouts  in  proposals  and  studies  which  include  all  uf  the 
factors  and  elements  mentioned  herein  and  that  the  breakout  should  be 
understandable  to  the  procuring  agency. 

It  is  no  doubt  true  that  when  developing  cost  data  on  a  specific  weapon 
system,  a  more  inclusive  presentation  can  be  made  than  is  defined  in  a 
general  way  in  this  section. 

B.  THEORY  AND  PRACTICE 

Before  discussing  methodology  for  estimating  costs,  it  might  be  well 
to  discuss  the  philosophy  on  which  the  task  group  based  its  approach.  Many 
take  the  approach  that  costs  are  a  great  unknown  factor,  and  that  it  is  im¬ 
possible  to  make  acceptable  and  useful  cost  predictions  of  future  weapon 
systems. 

We  refuse  to  accept  this  philosophy.  The  business  of  estimating  costs 
is  not  like  coping  with  an  unknown  phenomenon  in  the  area  of  physics  of  in 
the  sciences.  For  example,  there  are  fundamental  gaps  in  man's  under¬ 
standing  of  particles  in  the  atomic  nucleus  and  hence  man's  methods  of 
predicting  and  controlling  the  nuclear  forces  are  often  rudimentary  and 
approximate.  But  this  stumbling  block  is  a  real  one.  namely,  lack  of  fun¬ 
damental  knowledge.  This  is  certainly  not  the  case  with  costs  for  existing 
commercial  systems,  as  witness  the  detailed  cost-knowledge  of  automobiles 
and  television  sets  available  in  the  appropriate  industries,  nor  is  it  neces¬ 
sarily  the  case  with  the  military  systems. 

A  particular  aggregation  of  existing  hardware  has  a  finite  number  of 
pieces.  The  only  reason  it  may  be  difficult  to  ascertain  the  costs  of  a  svs- 
tem  in  being,  like  a  B-52  or  an  Atlas  missile,  is  that  pertinent  data  have 
probably  not  been  kept  in  a  form  suitable  for  conventional  cost  estimating 
purposes.  Of  course,  the  costs  of  futu re  systems  presents  uncertainty, 
especially  if  they  depend  ipon  new  technological  methods. 


The  major  sources  of  uncertainty^  in  cost  analysis  of  future  system 
and  force  structure  proposals  may  be  classified  in  various  ways.  Here,  we 
somewhat  arbitrarily  select  two  major  headings:  requirements  uncertainty 
and  cost-estimating  uncertainty.  Requirements  uncertainty  refers  to  varia¬ 
tions  in  cost  estimates  stemming  from  changes  in  the  configuration  of  the 
system  or  force  being  costed.  Cost -estimating  uncertainty  refers  to  varia¬ 
tions  in  cost  estimates  of  a  system  or  force  where  the  configuration  of  the 
system  or  force  is  essentially  constant.  These  latter  variations  arise  for 
numerous  reasons  --  e.  g. ,  differences  in  individual  cost  analysts,  errors 
in  basic  data  used  in  cost  analysis,  errors  in  cost-estimating  relationships 
used  in  making  cost  estimates,  and  so  on. 


Requirements  uncertainty  is  considered  first,  because  all  the  empirical 
stvdies  suggest  rather  conclusively  that  requirements  uncertainty  is  the 
major  source  of  uncertainty  in  cost  analysis  of  military  system  and  total 
force  structure  proposals.  Let  us  attempt  to  get  a  better  fix  on  what  is 
meant  by  "requirements  uncertainty."  Suppose  that  early  in  the  develop¬ 
ment  program  a  total  system  cost  estimate  is  developed  for  a  certain  sys¬ 
tem,  using  the  best  information  and  data  available  at  that  time.  Then  sup¬ 
pose  that  some  years  later  when  the  system  is  being  initiated  into  the  active 
inventory,  a  new  system  cost  estimate  is  made,  using  all  the  information 
and  data  available  at  that  time.  Typically  we  find  that  the  original  estimate 
is  less  than  the  second  one  and  the  difference  may  be  quite  large.  What  is 
the  reason  for  this  ?  Two  factors  immediately  come  to  mind:  (1)  The  price 
level  may  have  increased  during  the  time  interval;  (2)  The  quantity  (size  or 
force)  of  the  system  may  be  larger  in  the  second  estimate. 

These  two  factors  do  in  fact  typically  play  a  significant  role.  However, 
even  after  adjusting  for  them  --  i.  e. ,  putting  the  two  estimates  on  a  com¬ 
parable  basis  with  respect  to  price  level  and  quantity--the  inequality  still 
remains  in  many  instances.  .Studies  of  this  phenomenon  in  the  past  indicate 
that  the  ratio  (adjusted)  may  be  1.  5,  2,  3  or  sometimes  even  higher.  These 
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studies  also  indicate  that  the  primary  reason  for  this  is  that  the 
configuration  of  the  system  has  changed,  and  that  these  configuration 
changes  tend  to  be  cost-increasing  in  their  -ffect.  In  other  words,  the 
subject  being  costed  initially  is  often  markedly  different  from  that  costed 
later.  Here  system  configuration  change  means  change  in  hardware  charac¬ 
teristics  and/or  change  in  system  operational  concept  (soft  vs  hard,  fixed 
vs  mobile,  concentrated  vs  dispersed,  low  alert  vs  high  alert,  low 
activity  vs  high  activity  rate,  etc. ). 

Numerous  reasons  may  be  given  for  changes  in  system  configuration. 
The  following  are  a  few  examples: 

1.  With  respect  to  the  system's  hardware,  the  original  design  may 
fail  to  produce  the  desired  performance  characteristics,  and  as  a  result, 
the  hardware  configuration  has  to  be  changed.  Or,  sometimes  performance 
characteristics  themselves  may  be  changed  (upward)  with  a  resultant  change 
in  hardware  specifications  and  hence  cost.  Another  possibility  is  that  an 
attempt  may  be  made  to  get  the  system  sooner  than  was  originally  intended 
by  substituting  resources  for  time. 

2.  A  change  in  system  specifications  may  be  induced  purely  by  errors 
of  omission  in  establishing  requirements  initially  for  some  part  of  the  sys¬ 
tem.  For  example,  in  the  early  phase  of  the  ICBM  program  this  happened 
with  respect  to  the  ground  support  equipment  (GSE).  Correction  of  the 
error  led  to  rather  marked  changes  in  system  GSE  requirements,  and 
hence  to  an  increase  in  GSE  cost. 

3.  A  change  like  that  mentioned  in  Example  2  above  may  have  an  in¬ 
direct  effect  on  other  parts  of  the  system.  It  is  possible  that  personnel  re¬ 
quirements  may  be  changed.  Also,  an  item  like  personnel  requirements 
often  is  very  sensitive  to  changes  in  system  operational  concept  (e.  g.,  de¬ 
gree  of  system  dispersal,  alert  capability,  etc.  ). 

4.  The  strategic  situation  may  change.  This  may  lead  to  a  respecifi¬ 
cation  of  hardware  performance  characteristics.  Or,  even  if  the  hardware 
is  not  affected,  the  method  of  deploying  and  employing  the  system  may  have 
to  be  changed.  For  example,  to  reduce  vulnerability  of  the  system  to  sur¬ 
prise  attack,  a  higher  degree  of  dispersal,  hardness,  or  alert  capability 


may  be  required  to  meet  the  new  strategic  situation.  The  impact  of  such 
changes  on  system  installations  and  personnel  requirements,  to  cite  two 
examples,  is  obvious.  A  re-evaluation  of  the  strategic  situation  may  pro¬ 
duce  changes  in  system  force  size  (number  of  units  to  be  procured  for  the 
operational  force),  or  in  some  cases  a  change  in  the  number  of  years  the 
system  is  planned  to  be  kept  in  the  operating  inventory.  Both  of  these 
situations  may  be  regarded  as  a  form  of  requirements  uncertainty,  result¬ 
ing  in  a  substantial  impact  on  total  system  cost. 

The  above  are  only  illustrative  examples  of  a  few  of  the  many  reasons 
the  configuration  of  a  system  may  change.  But  the  key  point  is  that  require¬ 
ments  uncertainty  can  lead  to  wide  variations  in  total  system  cost,  even  in 
the  complete  absence  of  cost -estimating  uncertainty  (if  this  were  possible). 
There  is  such  a  thing  as  cost-estimating  uncertainty,  of  course,  but  as 
mentioned  previously,  it  tends  to  be  small  relative  to  requirements  uncer¬ 
tain?-/. 

Too  often  cost  elements  have  been  lumped  and  presented  in  a  cumulative 
form  before  the  analyst  begins  his  work.  It  is  then  not  possible  to  answer 
questions  about  the  significance  of  training  costs  alone,  or  of  R  It  D  costs 
alone,  etc.  For  purposes  of  advanced  planning,  cost  elements  must  be  kept 
in  an  easily  identifiable  form  as  explained  below.  Data  in  the  p  re -cumulated 
form,  perhaps  on  punched  cards  or  tape,  can  be  analysed  from  different 
viewpoints  to  answer  varying  questions  and  to  compare  the  same  cost  sub¬ 
headings  across  weapon  systems.  The  resulting  cost -data  frank  will  then 
be  a  fruitful  source  of  present  and  future  data  for  cost-effectiveness 
decision-making.  The  Air  Force  has  not  fully  required  keeping  of  past 
program  cost  data  in  a  form  appropriate  to  decision-making  heretofore. 

This  failure  jeopardises  the  ability  to  make  good  future  procurement 
decisions. 

There  must  also  be  uniformity  in  the  major  data  categories  which  are 
recorded,  as  well  as  in  the  definitions  of  those  categories.  Cost  estimating 
techniques  must  then  be  established  to  help  estimate  the  costs  under  each 
of  the  major  sub-headings  which  are  defined. 
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In  generating  pre-procurement  parametric  cost  data,  items  such  as 
the  following  must  be  given  specific  consideration: 

1.  Identify  cost  categories  covering  all  sources  of  major  system  cost. 
The  cost  of  a  booster  vehicle,  for  example,  goes  far  beyond  the  direct  pro¬ 
curement  and  launch  costs  of  the  booster.  There  are  costs  of  training  pro¬ 
grams,  logistic  support,  manufacturing  facilities,  tooling,  GSE,  etc.  For 
maximum  Air  Force  management  and  decision  visibility,  a  thorough  cost 
breakout  is  needed  and  costs  must  be  tabulated  under  numerous  standard 
sub-headings. 

2.  Costs  in  various  units  are  needed  since  several  resource  constraints 
exist  and  confront  the  Air  Force.  For  example,  the  "cost"  of  a  new  weapon 
system  is  constrained  in  terms  of  the  availability  of  man-years  of  critical 
scientific,  engineering,  and  technician  effort  needed  to  develop  and  operate 
the  system.  Also,  the  number  of  man-years  of  various  military  support 
and  command  skill  levels  needed  to  operate  the  system  must  be  considered 
to  be  a  constraint.  The  numbers  of  tons,  gallons,  or  other  units  of  critical 
materials  (e.  g. ,  amounts  of  fissionable  material  or  cubic  yards  of  concrete 
for  bases,  or  tons  of  light  metals,  etc. )  needed  to  develop  and  operate  the 
weapon  system  must  be  presented.  Suppose  that  in  1970  a  new  ballistic 
missile  system  is  to  be  procured.  It  would  be  helpful  ther  to  have  the  his¬ 
torical  data  on  the  amount  of  engineering  man-hours,  manufacturing  hours, 
critical  materials,  etc.,  which  were  necessary  to  develop  the  Atlas,  the 
Titan,  the  Minuteman,  and  other  missile  families. 

Even  the  influence  of  the  proposed  weapon  system  procurement  on  other 
industries  may  be  helpful.  In  the  early  fifties,  the  Air  Force  was  consider¬ 
ing  development  of  Atlas  and  Titan  1.  The  need  for  expansion  of  liquid  oxy¬ 
gen  production  if  these  systems  were  to  oe  developed,  tested,  and  operated 
was  a  "cost"  whose  projection  was  vital  to  full  cost-effectiveness  under¬ 
standing  of  the  importance  of  these  systems. 

3.  Costs  in  each  of  the  cost  units  defined  in  paragraph  2  must  be 
calculated  per  year  as  well  as  in  total.  The  reason  for  this  is  that  a  given 
weapon  system  in  total  cost  units  (of  critical  manpower  skills,  critical 
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materials  units,  or  in  dollars)  may  satisfy  the  total  resource  level 
available,  but  the  annual  expenditures  may  exceed  a  smooth  budget  pattern 
during  certain  years.  This  becomes  a  serious  problem  to  the  funding  and 
man-loading  of  other  weapon-system  development  programs  which  are  pro¬ 
ceeding  simultaneously.  Critical  manpower  peaks  and  valleys  are  to  be 
avoided. 

4.  Cost  sensitivity  and  coat  variance  considerations  should  be  in¬ 
cluded  in  the  presented  coat  data.  (See  Section  VII  on  risk  and  uncertainty. ) 
A  cost  estimate  standing  alone  on  the  page  of  a  report  does  not  inform  the 
reader  as  to  its  firmness  relative  to  other  cost  data  presented  on  other 
pages  of  a  study.  Thus  some  indication  of  the  precision  of  the  estimate 
should  be  given  with  the  cost  data,  and  the  sensitivity  of  final  results  to  the 
cost  uncertainties  should  be  portrayed.  Thus,  in  some  way  the  more 
tenuous  cost  estimates  must  be  highlighted  so  that:  (a)  the  proper  confi¬ 
dence  in  the  ultimate  cost-effectiveness  number  can  be  conveyed  to  the 
reader;  and  (b)  remedial  effort  and  perhaps  recommended  testing  can  be 
undertaken  by  the  Air  Force  to  further  identify  and  refine  the  data.  The 
latter  will  lead  to  better  cost  information  on  the  technological  area  in 
question  and  thus  future  cost-effectiveness  studies  will  be  more  valid. 


C.  A  GENERAL  COST  METHODOLOGY 

1.  Identify  the  various  critical  cost  commodity  categories: 


1 


U 


m. . 
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man-years  of  effort  of  type  i  needed  in  year  j  to  develop, 

make  operational,  and  operate  the  proposed  weapon  system 
or  major  weapon  subsystem. 

tons,  pounds,  yr.rds,  or  other  units  of  c rit'cal  material 
of  type  i  needed  in  yea.*  j  to  develop,  make  operational, 
and  operate  the  proposed  weapon  system  or  major 
weapon  subsystem. 


Examples  of  types  i  are:  fissionable  materials,  fuels  (of  given  type), 
concrete,  machine  tools,  heavy  metals  (of  given  type),  light  metals  (of 


given  type),  etc.  Dollar  costa  may  be  obtained  by  arithmetically  operating 
(Dollars  in  year  j)  =  ($).  =  {!..  x  ($/unit 


j)  =  ($)j  =  ^fij  X  ($/unit  lij)J 

+  )lm.. 
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x  ($/unit  m.j) 


(26) 


2.  In  order  to  estimate  the  amounts  of  1.^  and  and  hence  tl.e  dollars 
needed  annually  over  the  life-time  of  a  weapon  system  it  is  useful  to  develop 
a  cost-breakout  chart  of  the  type  shown  in  Figures  7,  8  and  9.  Note  that  the 
estimation  of  facility  sizes,  for  example,  the  size  of  the  manufacturing 
facility,  etc. ,  is  dependent  on  the  total  force  level  of  the  weapon-system 
procurement.  The  learning  curve  effects  are  also  dependent  on  this  force 
level.  For  these  reasons,  the  estimating  of  the  1^.  costs,  the  m^  costs, 
and  the  total  yearly  dollar  costs  should  be  done  parametrically  for  various 
planned  force-level  procurements  of  the  weapon  system. 


The  effect  of  force  size  on  some  constraining  "costs"  and  the  insensiti¬ 
vity  of  other  cost  elements  to  force  size  can  be  seen  in  Figures  10  and  11. 
Figure  10  shows  that  the  number  of  engineers  needed  to  develop,  test,  and 
operate  a  weapon  system  is  relatively  insensitive  to  force  size.  On  the 
other  hand,  the  dollars  per  year  for  all  items  of  manpower  and  materials 
(IOC  and  DOC)  may  depend  very  heavily  on  the  force  size  procured,  as 
shown  in  Figure  1 1.  Note  that  RID  costs  are  the  same  for  the  two  planned 
force  levels  in  Figure  11. 


D.  COST  ESTIMATING  RELATIONSHIPS  (CER^ 

A  functional  expression  which  states  that  the  cost  of  someti  Ing  may  be 
estimated  on  the  basis  of  a  certain  variable  or  s.*t  of  variables  is  called  a 
Cost  Estimating  Relationship  (CER).  These  expressions  may  be  simple  or 
complex  in  terms  of  functional  form  and/or  the  number  of  variables  taken 
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For  "sisinf "  facilities, 
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procurement  level: 
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For  constructing  learning 
curves,  parts  inventories, 
etc.,  input  weapon 
system  forces  level: 

Number  of  bases 
Number  of  units 
Units  per  year 


FIGURE  7 


COST  BREAKOUT  CHART  FOR  ESTIMATING  CRITICAL  RESOURCE  UNITS 
REQUIRED  TO  DESIGN.  DEVELOP.  AND  OPERATE  A  WEAPON  SYSTEM 


ASPECTS  OF  INDIRECT  OPERATING  COSTS 


into  account.  Many  Cost  Estimating  Relationships  (CER's)  such  as  that 
shown  in  Figure  12  exist  in  the  literature.  Excerpts  from  Headquarters 
AFSC  Cost  Estimating  Relationship  Program  Plan  dated  31  July  1962  are 
included  in  this  Section  D.  These  CER's  are  helpful  in  generating  cost 
estimates  for  the  many  cost  categories  of  Figures  7,  8  and  9. 


$/lb 

Thrust 


TYPICAL  COST  ESTIMATING  RELATIONSHIP  (CER) 
BASED  ON  PAST  PROGRAM  DATA 


The  process  of  estimating  i%  by  definition, approximate.  The  variability 
of  an  estimate  will  be  relatively  small  if  the  item  being  costed  is  essentially 
the  same  in  the  future  as  it  was  in  the  past.  For  example,  for  the  cost  of 
flying  a  B-47  from  New  York  to  Los  Angeles,  the  number  in  the  crew,  the 
pay  per  person,  and  the  B-47  fuel  costs  per  hour  are  established  and  can  be 
expected  to  stay  essentially  the  same,  assuming  a  constant  dollar.  The 
variability  will  also  be  small  if  the  item  being  costed  varies  in  relation  to 
some  predictable  parameter.  For  example,  the  cost  of  a  landing  strip 
varies  with  the  number  of  cubic  yards  of  concrete,  even  if  a  projected 
landing  strip  is  much  longer  and  thicker  than  any  existing  strip. 

The  costing  problem  becomes  much  more  complex  ir.  the  case  of  a 
weapon  system  or  supporting  system  that  is  greatly  different  from  any  sys¬ 
tem  on  which  actual  cost  data  are  available.  Since  historical  costs  are  the 
only  basis  for  projecting  future  costs,  some  historical  relationship  must  be 
found.  If  the  analyst  had  time  to  break  the  future  system  down  into  its  most 


detailed  elements,  we  would  find  that  many  of  the  detailed  elements  were 
similar  to  elements  in  earlier  systems  and  could  be  related;  e.  g. ,  cost  of 
micro-circuit  elements. 

It  is  often  possible  to  estimate  costs  by  some  broader  aggregation  than 
micro-circuits  at  the  component  or  even  the  subsystem  level,  based  on 
some  parameter  of  performance  or  specification  which  has  a  reasonable 
correlation  with  cost.  For  discussion  purposes,  the  correlation  between 
that  parameter  and  cost  is  referred  to  as  the  Ccst  Estimating  Relationship 
(CER)  for  that  element  of  cost.  A  CER  can  also  be  identified  for  non-hard- 
ware  cost  elements,  such  as  fuel  and  lubricant  costs  as  a  function  of  flying 
hours  or  depot  maintenance  might  be  given  as  a  per  cent  of  acquisition  cost. 
Obviously,  the  broader  the  cost  element  for  which  a  CER  can  be  established, 
the  less  time  is  required  to  compute  the  total  cost  estimate. 

The  Cost  Analyst  should  not  wait  until  he  needs  a  CER  to  collect  histor¬ 
ical  data,  determine  the  moat  appropriate  correlation,  and  document  the 
CER.  This  is  a  time-consuming  effort  and,  at  least  for  the  more  obvious 
elements  of  cost,  a  CER  should  be  anticipated  and  established  between 
specific  coat-estimating  assignments.  Each  assignment  will  undoubtedly 
reveal  a  requirement  for  still  ether  CER's  that  are  not  available,  but  the 
time  required  can  be  minimised  by  proper  planning  and  oreparation  of  the 
CER. 

CER's  should  be  prepared  for  different  levels  of  aggregation  (in  equip¬ 
ment  cost,  for  example,  not  only  at  the  subsystem  level  but  also  at  the 
component  level)  where  possible  and  appropriate  so  that  the  cost  analyst 
has  some  flexibility  in  their  use.  Depending  on,  first,  the  degree  of  detail 
to  which  a  system  can  be  described  for  the  cost  analyst  and,  second,  the 
time  available  to  prepare  the  estimate,  the  cost  analyst  can  then  use  the 
broader  CER  (involving  leas  computation  time;  possibly  less  accurate)  or 
the  finer  CER  (involving  more  computation  time;  possibly  more  accurate). 
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E.  DEVELOPING  CER's 


A  Cost  Estimating  Relationship  is  simply  a  functional  expression  which 
states  that  the  cost  of  something  may  be  estimated  on  the  basis  of  a  certain 
variable  or  set  of  variables  (see  Figure  12).  These  expressions  may  be 
simple  or  complex  in  terms  of  functional  form  and/or  the  number  of  variables 
taken  into  account.  The  simplest  possible  case  is  where  the  cost  of  some¬ 
thing  may  be  estimated  almost  entirely  as  a  function  of  a  single  variable. 
Examples  are:  "Cost  per  flying  hour,"  "Cost  per  pound  structure,"  "Cost 
per  bit  of  memory,"  etc.  At  other  times  more  complicated  multivariate 
functions  of  various  mathematical  types  may  be  used  in  CER's.  Also,  it 
should  be  pointed  out  that  a  cost  estimating  relationship  need  not  necessarily 
be  "continuous."  It  may  be  discontinuous,  or  a  "step  function." 

In  any  event,  a  cost  estimating  relationship  expresses  cost  as  a  function 
of  a  variable  or  variables.  The  fundamental  steps  in  developing  cost  esti¬ 
mating  relationships  are  as  follows: 

1.  Determine  what  these  variables  are.  This  is  easier  said  than  done. 
There  is  no  known  method  to  readily  identify  what  the  key  variables  are. 

The  graphical  fitting  appraoch,  together  with  discussions  with  appropriate 
engineering  and  manufacturing  personnel,  seems  to  be  the  only  method  of 
determining  the  significant  variables.  Success  in  identifying  the  variables  is 
largely  dependent  upon  the  ingenuity  of  the  analyst  and  the  amount  of  effort 
and  time  spent  in  exploring  the  various  combinations  of  all  identifiable  para¬ 
meters.  The  simplest  variables  such  as:  cost  related  to  quantity,  cost 
related  to  weight,  cost  related  to  size,  ratios,  etc. ,  probably  should  be 
tried  first.  Use  of  percentage  relationships  are  also  helpful.  For  example, 
on  rocket  engine  pumps  past  data  may  show  that  the  costs  are:  labor  -  45 
per  cent,  material  -  33  per  cent,  overhead  -  22  per  cent. 

2.  Determine  an  appropriate  functional  form.  The  representation  of 
how  cost  will  vary  in  relationship  to  some  variable  is  a  cost  pattern,  or  cost 
behavior.  A  cost  pattern  maybe  shown  graphically  or  possibly  expressed 
algebraically.  Plotting  a  graphic  picture  ot  the  data  is  a  way  to  find  a  suit¬ 
able  functional  form  and  to  assess  and  determine  the  variability  of  costs. 
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3.  Estimate  the  numerical  value  of  the  parameter*  in  the  functional 
form.  In  using  a  visual  method  of  averaging  a  curve  through  the  data  plot 
points,  there  would  be  little  probability  of  two  persons  drawing  the  same 
curve.  It  would  be  desirable  to  have  a  method  where  the  same  curve  would 
be  consistently  obtained  from  the  same  data.  This  can  be  accomplished  by 
the  method  of  least  squares  or  other  commonly  used  curve-fitting  procedures. 

4.  Where  possible,  give  some  indication  as  to  the  confidence 
associated  with  the  estimating  relationships.  In  a  statistical  sense,  this 
may  be  accomplished  by  .furnishing  standard  errors  of  estimate,  confidence 
intervals,  and  the  like.  Where  this  is  not  possible,  less  formal  (perhaps 
non-quantitative)  statements  may  be  used. 

5.  If  the  relationships  hold  only  for  certain  ranges  of  the  "cost- 
generating"  variables,  these  ranges  should  be  stated.  Stating  the  problem 
in  five  straightforward  steps  like  1-5  above,  may  convey  the  impression 
that  the  task  of  developing  usable  cost  estimating  relationships  is  a  rela¬ 
tively  simple  process.  This  is  definitely  not  the  case,  especially  in  the 
field  of  advanced  weapon-system  cost  analysis.  A  fundamental  problem  {» 
the  paucity  of  meaningful  quantitative  data.  And  even  when  data  are  avail¬ 
able,  the  number  of  observations  (the  "sample  sise")  is  likely  to  be  small. 

F.  AFSC  FORMAT  FOR  RECORDING  CER's 

Careful,  concise  documentation  is  essential  in  establishing  a  reliable 
cost  estimating  relationship.  Ideally,  it  should  be  possible  for  any  analyst 
to  retrace  from  beginning  to  end  the  specific  steps  originally  taken  in  de¬ 
riving  each  cost-estimating  relationship.  More  important,  however,  is 
that  the  documentation  would  enable  another  analyst  to  decide  intelligently 
whether  a  given  CER  can  be  used  reliably  in  costing  a  particular  system  to 
solve  a  particular  problem.  Even  though  the  analyst  has  little  confidence  in 
the  CER,  he  should  document  it  in  the  manner  prescribed  so  that  there  is  a 
specific  record  aud  a  basis  for  further  research.  He  will  thereby  save  time 
for  himself  and  others  in  researching  the  same  sources  again.  Further,  it 
v  ill  identify  a  problem  area.  Figure  13  is  a  possible  format  (as  used  by 
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(Security  Classification) 

COST-ESTIMATING  RELATIONSHIP  PROGRAM 


(Code) 

(Date) 

Subject : _ _ 

(System  &  Subsystem  for  which  cEr  is  Developed) 

CER  Formula  (and  graph  is  appropriate)  Derived: 

Definitions  Used: 


Assumptions: 


Data  Sources  Used  in  Developing  CER: 


Specific 

Item 

mnSSw 

Cost 

Year 

•‘PaTno 

MB  WM  i  P  H  Tff  S  MM  ■■■ 

(Examples  of  nonfinancial 
parameters:  weight,  sise, 
thrust,  range,  materials, 
rate,  RPM,  output,  power, 
time,  etc. ) 

Developed  by: 

(Organisation) 


Statistical  Techniques  and  Methods  Used: 
(Including  Cost  Adjustments) 


Degree  of  Confidence  and/or  Limitation*:  Other  Comments: 


FIGURE  1  L  AF SC.  FORMAT  FOR  RECORDING  CER’s 


I 


# 


AFSC)  upon  which  to  start  documentation.  It  is  not  to  be  confined  to  one 
page,  as  the  CER  itself  will  dictate  the  amount  of  detail  required. 

Codes  may  follow  the  areas  of  systems,  subsystem  and  components  as 
follows : 


1st  Digit  2nd  Piatt 


Aircraft 

- 

1 

Research  -  1 

Missile 

- 

2 

Development  -  2 

Electronics  - 

3 

Investment  -  3 

Space 

- 

4 

Operating  -  4 

3rd  V  4th  Digits 

Airframe 

-  01 

Data  Processing 

08 

Propulsion 

-  02 

Communications 

09 

Guidance  It  Control 

-  03 

Equipment 

10 

Missile  Launcher 

-  04 

Manpower 

11 

Facilities . 

-  05 

Equip  It  Install  Replace  - 

12 

Re-entry 

-  06 

Training 

13 

Data  Acquisition 

-  07 

System  Maintenance 

14 

A  CER  for  development  of  a  Communication  Space  Set  ell  it  e  would  be  coded 
4*2-09.  This  will  then  permit  a  minimum  of  four  major  classifications 
upon  which  to  file  all  CER's,  even  though  the  level  of  the  CER  might  be  at 
the  very  lowest  component  category. 


C.  THE  FUTURE  DEVELOPMENT  OF  COSTING 

Many  companies  have  already  developed  computerised  cost  models 
which  utilise  breakdowns  of  the  type  shown  in  Figures  7,  8  and  9,  using 
hundreds  of  cost  relationships  (CER's)  of  the  type  shown  in  Figure  12.  The 
National  Aeronautics  and  Space  Administration  (Future  Projects  Office, 
MSFC,  Huntsville,  Alabama)  has  developed  an  IBM  7094  Launch  Vehicle 
Cost  Model  which  can  generate  annual  costs  in  the  various  categories;  R  & 

D,  Facilities,  Direct  Costs,  etc.,  for  launch  vehicles. 
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Another  aspect  of  the  cost  estimation  problem  is  sensitivity  to  inputs. 
Too  often  th»re  is  a  refusal  to  estimate  costs  because  the  data  inputs  are 
not  precise  or  accurate.  An  estimate  of  the  variability  of  (1)  over -all  costs 
per  year,  and  (2)  total  program  costs  can  be  made  by  using  the  cost  variance 
concept,  a  PERT -type  procedure,  in  which  each  cost  element  estimate  is 
inserted  in  three  forms:  least  possible,  expected,  and  maximum  possible 
cost  (see  BIBLIOGRAPHY,  "Cost  Variance:  A  Proposed  New  Cost 
Prediction  Method"). 


SECTION  VI 


EFFECTIVENESS  FACTORS  FOR 
COST-EFFECTIVENESS  MODELS 
(TASK  8.  2) 


A.  INTRODUCTION 

In  Section  IV  the  reader  was  presented  several  possible  cost-effective¬ 
ness  models.  In  each  model,  it  was  essential  to  develop  effectiveness 
measures  and  cost  factors  in  order  to  arrive  at  cost-effectiveness  values. 
Section  V  presented  cost  estimating  procedures.  The  objective  of  this 
section  is  to  outline  the  ingredients  for  a  better  evaluation  of  effectiveness. 
The  factor  so  identified  will  enable  one  to  insure  that  all  essential  factors 
are  included  in  effectiveness  measures.  The  Task  Group  II  Report  presents 
more  detailed  methodology  appropriate  for  the  measurement  of  these  effec¬ 
tiveness  elements. 

B.  EFFECTIVENESS  FACTORS 

Effectiveness  is  a  far-ranging  and  comprehensive  concept  when  applied 
to  military  weapon  systems.  For  example,  an  individual  ballistic  missile 
system  can  be  considered  "effective"  only  if  it  can  satisfy  a  rather  long  list 
of  requirements.  Typically,  the  list  will  include:  (1)  be  in  a  launch-ready 
condition  when  the  order  is  given  to  fire;  (2)  decode  and  accept  the  launch 
signal;  (3)  go  through  a  prc -launch  verification  sequence;  (4)  launch  and 
fly  a  planned  trajectory;  (5)  stage  one  or  more  times;  (6)  cut  off  the  engine 
when  proper  velocity  and  direction  are  attained;  (7)  separate  the  payload, 
arm,  and  fuse;  (8)  arrive  within  a  specified  distance  of  ground  aero;  and 
(9)  detonate  the  warhead  at  the  correct  altitude  with  (iu)  a  specified  mini¬ 
mum  yield. 

Because  of  this  proliferation  of  requirements,  we  have  chosen  to  con¬ 
solidate  t'..e  elements  of  effectiveness  somewhat.  We  will  discuss  them 
here  more  nearly  in  the  context  of  Secretary  McNamara's  description  of 
"dependat  lity'  (applied  to  both  missiles  and  aircraft)  which  he  defined  as 


the  product  of  Availability,  Survivability,  Reliability,  and  Penetrability.-^ 
To  this  list  »e  will  add  Lethality,  and  we  will  treat  Launch  Reliability  and 
Mission  Reliability  separately. 


The  effectiveness  elements  for  an  aircraft  are,  for  the  main  part, 
essentially  the  same  as  those  of  a  missile--that  is,  the  system  has  to  be 
ready,  survive  any  enemy  attack,  reach  the  target  area,  penetrate  enemy 
defenses  and  deliver  a  lethal  blow.  An  additional  requirement  unique  to 
aircraft  systems  is  the  safe  return  to  a  friendly  base.—  For  ground  equip* 
ment,  like  radarsr^  or  L-systems  (Command  and  Control),  the  effective¬ 
ness  elements  are  similar,  provided  one  substitutes  performance  measures 
for  penetration  and  lethality. 

We  will  attempt  to  show  here  how  each  of  these  effectiveness  elements 
is  influenced  by  one  or  more  operational  factors.  Then  we  will  give  some 
examples  of  how  certain  individual  or  collective  elements  can  be  quantified 
through  the  application  of  mathematical  models,  and,  where  possible,  show 
some  applications  to  actual  systems. 


C.  AVAILABILITY 

The  first  effectiveness  element  to  be  discussed  here  is  Availability, 
which  has  also  been  referred  to  in  the  past  (not  always  accurately)  as 
Operational  Availability,  Operational  Readiness,  Alert  Readiness,  Ready 
Rate,  and  Real  In-Commission  Rate.  It  can  be  defined  as  the  probability 
that  the  system  will  be  in  a  good  or  usuable  condition  at  some  randomly 
chosen  time  in  the  future. 
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"  Statement  of  Secretary  of  Defense  Robert  S.  McNamara  before  the  House 
Armed  Services  Committee  on  the  Fiscal  Year  '65-'69  Defense  Program 
and  the  1965  Defense  Budget,  January  27,  1964.  The  Task  Group  II  Report, 
Vol.  Ill,  adds  propellant  depletion  probability,  command  and  control  pro¬ 
bability,  guidance  accuracy,  and  targeting  policy  to  this  list. 

^  See  Example  A,  Vol.  Ill,  of  the  Task  Group  II  Report. 

—  See  Example  C,  Vol.  Ill,  of  the  Task  Group  II  Report. 


A  system  may  be  unready  to  launch  because  of  scheduled  or  unscheduled 
maintenance  activities,  periodic  inspection,  crew  training,  or  other  activities. 
Even  if  not  in  one  of  these  unready  states,  the  system  may  fail  to  countdown 
successfully  because  of  an  existing,  but  previously  undiscovered  defect.  In 
the  latter  case,  the  deficiency  may  have  been  inadvertently  ascribed  alterna- 
tively  to  Availability  (we  thought  it  was  ready,  but  it  really  wasn't),  or  to 
Launch  Reliability  (the  lights  were  green,  but  the  countdown  was  NO-GO). 

In  those  instances  where  a  function  is  neither  checked  periodically  nor  mon¬ 
itored  continuously,  we  will  interpret  a  ..-feet  or  failure  of  the  function  as  a 
contribution  to  unreliability  of  launch  or  flight. 


Mathematically,  long-run  or  steady -state  availability  can  be  expressed 


MTBI 


where  MTBI  is  the  mean-operating-time-between-interruptions,  for  any 
cause,  and  MTTR  is  the  mean  duration  of  those  interruptions.  Thus,  MTBI 
has  the  connotation  (though  not  always  the  precise  meaning)  of  mean -time  - 
between  failures.  MTTR  has  the  sense  of  a  mean-time-to-repair,  which  we 
will  generalize  to  support  capability.  Improvements  in  either  alert  reli¬ 
ability  or  support  capability  (i.  e.,  higher  MTBI  or  lower  MTTR)  result  in 
improved  (increased)  availability. 

The  quantification  of  availability  through  this  simple  formula  is  seldom 
found  to  be  directly  useful  in  real-world  cases.  MTBI  can  only  be  interpre¬ 
ted  as  an  inherent  quality  in  the  special  case  of  continuous  (and  complete 
and  accurate)  monitoring.  MTTR  can  be  independent  only  where  there  is 
immediate  administrative  response,  and  continuous  availability  of  mainte¬ 
nance  resources.  Generally,  both  are  functions  of  several  interrelated 
parameters. 

Typical  of  actual  operations  are  periodic  (and  in  some  ways  imperfect) 
inspection  or  checkout, 'administrative  (or  "policy")  delay  in  maintenance 
response,  waiting  fo”  maintenance  resources  ("queuing")  and  an  imperfect 
quantitative  knowledge  concerning  the  system  parameters  which  determine 
operational  availability.  Several  examples  in  the  Technical  Supplement 
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(Volume  III  of  the  final  report  of  Task  Group  IV)  explore  the  subject  of 
inspection  policy,  support  policy,  and  testing  policy  as  they  may  affect  avail¬ 
ability  (and  system  effectiveness) through  their  effect  on  MTBI  and  MTTR. 

1.  Mean-Time -Between-Interruptions  (MTBI)  The  mean -t ime -between - 
interruptions  of  the  alert  status  of  a  weapon  system  is  a  measure  of  how 
long  the  system  can  be  expected  to  remain  in  a  status  of  real  (as  opposed  to 
"green  light")  readiness  before  that  status  is  terminated  by  scheduled  main¬ 
tenance  (checkout.  Technical  Order  compliance,  etc.),  unscheduled  mainte¬ 
nance  (repair  of  a  mission-critical  failure  discovered  through  continuous 
monitoring,  or  periodic  testing)  or  the  occurrence  of  an  initially  hidden 
mission-critical  failure.  (This  last  notion  gives  rise  to  the  distinction  be¬ 
tween  "real"  and  "green  light*  readiness.)  The  MTBI  is  affected  by  the 
reliability  state-of-the-art,  the  level  of  system  complexity,  the  maintenance- 
operations  plan  (continuous  monitoring  or  periodic  inspection,  with  repair  or 
replacement  as  necessary  or  periodic  mandatory  replacement  without  in¬ 
spection),  the  operational  (physical)  environment,  the  predictive  capability 

of  testing  procedures,  and  time,  as  it  involves  reliability  growth. 

2.  Support  Capability  (Mean-Time -To-Repair)  Support  Capability  is 

the  other  side  of  the  availability  coin;  some  refer  to  it  broadly  as  "logistics," 
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implying  the  traditional  military  supply-support  activities.—  We  will  inter¬ 
pret  it  here  as  MTTR,  the  time  required  to  restore  a  weapon  system  to  a 
truly  ready  cono,.ion,  whether  from  preventive  maintenance  (formerly 
"scheduled"  maintenance),  corrective  maintenance  (formerly  "unscheduled" 
maintenance),  or  other  non-ready  states.  As  such,  it  ic  evident  that  MTTR 
is  affected  by  anything  which  adds  time  to  the  process.  Some  more  impor¬ 
tant  examples  are:  (1)  the  quantity  of  maintenance  ground  equipment 
(waiting  caused  by  shortager);  (2)  snares  level  and  location;  (3)  depot  repair 
capability;  (4)  transportation:  (5)  maintenance  manpower  skills  and  numbers; 
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—  Others  use  the  term  "maintainability,"  which  we  interpret  here  more 

restrictively  as  the  quality  repa irabilitv  as  exemplified  through  "human 
engineering . " 


and  (6)  maintenance  management  policies.  In  the  Technical  Supplement 
mentioned  above,  there  is  an  example  of  how  Maintenance  Ground  Equip  - 
ment  (MGE)  and  manpower  provisioning  can  directly  affect  the  Mean  Time 
to  Repair. 

D.  SURVIVABILITY 

Survivability  is  the  probability  that  a  system  will  either  (1)  be  launched 
and  removed  from  the  threatened  environment  before  it  can  be  attacked  (as 
with  warning),  or  (Z\  "ride  out"  some  anticipated  initial  attack.  In  the  first 
instance  (applicable  primarily  to  manned  aircraft),  the  basic  parameters 
are  the  amount  of  reliable  warning  time,  and  the  reaction  time  from  "com¬ 
mit"  through  reaching  a  "safe"  environment.  In  the  latter  case,  which  is 
commonly  assumed  for  hard-site  ballistic  missiles,  many  things  can  be¬ 
come  important:  blast  hardness  (i.  e. ,  resistance  to  overpressure);  elec¬ 
tronic  hardness;  dispersal  mobility;  deception;  active  defense;  and  above  all, 
the  weight  or  severity  of  the  expected  attack.  If  a  weapon  system  is  intend¬ 
ed  to  provide  a  credible  deterrent  threat  for  a  sub  antial  time  (say  days  or 
even  weeks)  after  initiation  of  hostilities,  it  must  not  only  survive  possible 
missile  attacks,  but  perhaps  also  manned -bomber  attacks.  In  addition,  such 
extensive  periods  of  operation  must  be  supported  in  the  likely  absence  of 
"normal"  (i.e.,  peacetime)  services  like  commercial  power,  telephones, 
and  even  highway  travel. 

E.  RELIABILITY 

1.  Launch  Reliability  Launch  Reliability  is  the  probability  that  an 
"available"  system  (i.e.,  in-commission  and  having  no  hidden  defects  de¬ 
tectable  by  monitoring  or  periodic  checkout)  can  be  launched  on  command. 

In  general,  it  can  be  thought  of  as  being  made  up  of  Command  Reliability 
(receive,  interpret,  and  act  within  a  specified  time  period  upon  the  launch 
signal).  Countdown  Reliability  (accomplish  the  specified  pre-launch  proce¬ 
dures  within  the  allotted  time),  and  Initiation  Reliability  (perform  the  irre¬ 
versible  or  "one-shot"  sequence  of  launch  events  such  as  firing  squibs, 
door  oidnancc,  igniters,  etc.).  All  three  of  these  subelements  depend 
also  in  some  fashion  on  the  quality  of  maintenance  activities  accomplished 
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during  the  period  of  strategic  alert.  In  other  words,  they  are  affected  by 
the  same  things  as  MTBI,  though  not  necessarily  in  the  identical  manner. 
For  example,  the  pre-launch  procedures  are  often  similar  and  sometimes 
identical  to  periodic  exercises  performed  for  verification  of  alert  status. 

2.  Mission  Reliability  Mission  Reliability  (Flight  Reliability  for 
ballistic  missiles)  is  the  probability  that  the  system  will  execute  the  proper 
flight  sequence.  We  mean  to  include  performance  capability  (a  function  of 
the  design)  as  an  integral  part  of  this  mission  reliability.  For  example,  the 
failure  of  a  missile  to  reach  the  target  due  to  insufficient  fuel  capacity  is 
attributable  to  design  inadequacy  rather  than  "failure"  in  a  more  conven¬ 
tional  sense. 

For  ballistic  missiles.  Mission  Reliability  ordinarily  includes,  in 
addition  to  propulsion  and  control  and  other  factors,  proper  engine  cut-off, 
staging,  and  guidance.  For  manned  aircraft,  both  hardware  performance 
and  human  performance  (correct  navigation  to  target,  aiming,  and  weapon 
delivery)  are  involved.  For  an  ALQ-27  Jammer  System,  it  would  include 
detection  of  enemy  radiation,  selection  of  response  mode,  and  subsequent 
radiation  of  the  proper  jamming  signals.  Once  again,  the  hardware  reli¬ 
ability  is  related  to  the  quality  of  maintenance  in  the  ground  environment 
for  both  aircraft  and  missiles. 

F.  PENETRABILITY 

Penetrability  is  the  probability  that  a  weapon  system  will  survive  a 
defense  environment  and  arrive  at  the  target  intact.  For  manned  aircraft, 
this  probability  is  a  function  of  such  things  as  the  penetration  mode  (for 
example,  low  level  flight  to  avoid  detection),  speed,  maneuvers,  electronic 
countermeasures,  decoys,  etc.  For  ballistic  missiles,  for  example,  pene¬ 
trability  may  be  expected  to  be  100%  against  a  no-defense  environment, 
while  anti-ICBM  environments  make  penetration  aids  and  terminal  maneu¬ 
vers  important.  This  is  an  area  where  time  is  also  certain  to  be  a  factor; 
chronological  improvements  can  be  expected  in  the  quality  of  both  offensive 
and  defensive  tactics,  so  particular  levels  of  either  must  be  associated 
with  a  particular  point  in  time. 
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G.  LETHALITY 


Lethality  ia  the  probability  that  weapon  effects  will  destroy  the  target. 
For  a  point  target,  this  is  a  function  of  the  accuracy  of  delivery,  usually 
expressed  as  a  Circular  Error  Probable  (CEP),  and  the  lethal  radius 
(LR),  which  is  in  turn  a  function  of  warhead  yield,  burst  altitude,  and 
target  hardness.  For  area  targets,  lethality  can  be  related  to  these  same 
parameters  through  simple  nomograms,  which  can  give  (in  addition  to  sim¬ 
ple  probability  estimates)  the  expected  fraction  of  an  area  target  that  will 
be  destroyed.  (This  type  of  information  is  also  available  for  multiple  war¬ 
heads,  or  multiple  weapon  launches. )  The  latter  quantity  can  be  considered 
a  figure  of  merit  attributable  to  such  elements  as  yield,  aim  point,  CEP 
and  height  of  burst. 

H.  OTHER  CQKSffiJSBAIlOMS 

Certain  other  qualities  of  a  weapon  system  may  be  important  relative 
to  total  effectiveness,  even  if  their  impact  is  more  difficult  to  quantify. 

For  example,  the  ability  to  retarget  a  ballistic  missile  quickly  and  simply 
may  allow  a  reduction  in  the  extent  of  overlapping  coverage  for  high-priority 
targets,  and  thus  permit  either  an  improvement  in  the  long-term  coverage 
of  secondary  targets,  or  alternatively  a  reduction  in  the  required  sise  (and 
cost)  of  the  total  force  structure.  The  inherent  flexibilities  of  manned 
systems  are  likewise  significant,  even  if  hard  to  quantify. 

For  many  weapon  systems  safety  is  a  paramount  consideration.  Unless 
safety  features  are  carefully  considered  during  the  development  process, 
there  may  be  a  significant  probability  that  a  system  is  activated  by  error 
(operator,  maintenance,  spurious  signals,  failure  of  a  critical  circuit  or 
function,  etc.  )  Military  and  strategic  consequences  of  such  errors  are 
enormous,  and  their  prevention  is  frequently  an  overriding  tactor  in  the 
choice  of  its  design  and  configuration. 

For  some  systems  security  is  a  vital  'actor.  What  is  the  probability 
that  a  saboteur  could  take  over  a  system  »r.d  render  it  incapable  of  use,  or 
worie,  use  it  against  us1  Although  it  may  be  difficult  to  quantify  both  safely 


and  security,  there  can  be  r,o  question  that  system  design  and  operation 
criteria  must  reflect  a  thorough  assessment  of  these  real  probabilities. 


SECTION  VII 


RISK  AND  UNCERTAINTY  IN  COST-EFFECTIVENESS 

(TASK  8,  3) 

Buyers  and  designers  of  weapon  systems  face  two  types  of  future  events 

which  can  affect  their  plans.  One  is  risk;  the  other  is  uncertainty.  Both 

are  involved  in  the  expectations  of  the  future  stream  of  events.  Although  it 

may  be  conventional  to  class  the  results  of  all  future  events  which  can  lead 

to  degradation  of  mission  accomplishment  as  risk  and  uncertainty,  we  shall 

demonstrate  that  a  useful  and  realistic  distinction  can  be  made  between  the 

9/ 

two  concepts  from  the  effectiveness  as  well  as  from  the  cost  point  of  view.— 
A.  RISK 

Risk  refers  to  the  variability  of  outcome  which  is  quantitative  and 
measureable.  It  is  not  necessary  that  we  be  able  to  predict  the  outcome  of 
a  particular  event;  e.  g. .  that  a  specific  black  box  will  fail  within  a  specific 
period  of  time.  On  the  contrary,  it  is  only  necessary  that  the  probability  of 
outcome;  i.  e.,  rate  of  failure,  be  established  from  a  large  number  of  ob¬ 
servations. 

Probability  of  outcome  can  be  established  in  one  of  two  ways.  The 
a  priori  probability  of  outcome  can  be  established  when  the  characteristics 
are  known  beforehand;  the  empirical  probability  can  be  established  when 
there  are  enough  sample  observations  repeatedly  made.  In  both  cases,  we 
are  able  to  predict  the  statistical  probability  of  outcome  with  a  degree  of 
certainty  which  we  classify  as  risk.  The  key  here  is  that  the  likelihood 
(risk)  of  an  adverse  event  (leading  to  lower  effectiveness)  may  be  insured 
against  and  that  we  can.  therefore,  associate  a  cost  with  risk. 


—  Frank  Knight  is  generally  considered  the  originator  of  this  approach, 
ills  classic  treatment  Risk,  Uncertainty,  and  Profit,  published  originally 
in  l(*2  1  bv  Houghton  \* i ^f*l in  Co.  ,  is  now  in  its  8th  impression.  See 
Reference  ti  for  further  insight  on  Bavesian  approaches  to  tne  problem. 
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For  example,  the  cost  of  the  support  of  a  weapon  system  (logistics, 
maintenance,  etc.)  reflects  in  large  part  costs  arising  from  limitations  of 
an  engineering  nature  in  contrast  to  costs  arising  from  events  external  to 
the  system.  Based  upon  probabilities  of  failure,  repair  policies,  turn¬ 
around  times,  etc.,  an  expected  support  cost  may  be  determined.  If  we 
estimate  that  a  component  will  have  an  unacceptably  high  probability  of 
failure  during  the  mission,  redundancy  may  be  utilized,  or  more  effort  may 
go  into  reliability  development  of  that  component.  The  cost  of  risk  in  this 
latter  case  is  the  added  cost  of  redundancy  and/or  development.—^ 

In  the  case  of  risk,  it  is  not  difficult  to  handle  the  impact  on  decision 
making  and  use  of  resources.  Since  we  are  concerned  with  the  mean  or 
modal  outcome,  or  range  and  dispersion  of  outcomes,  any  possible  losses 
or  degradation  of  performance  can  be  incorporated  into  the  design  of  the 
system  and  added  to  the  cost  relative  to  the  added  level  of  protection. 

Risk  relates  to  the  phenomenon  which  creates  variability,  but  which  can 
be  overcome  (or  insured  against)  by  adding  to  costs.  It  does  not  preclude 
objective  decision  making  since  it  can  be  reduced  to  a  matter  of  cost.  It  is 
uncertainty  which  gives  rise  to  the  need  for  an  entirely  different  framework 
for  decision  making  and  resource  use. 

B.  UNCERTAINTY 

The  distinction  between  risk  and  uncertainty  is  that  the  latter  cannot  be 
established  empirically  or  quantitatively  but  instead  is  the  result  of  a  sub¬ 
jective  reasoning  process.  Uncertainty  always  exists  when  the  parameters 
of  the  probability  distribution  (for  example,  the  expected  availability  of  the 
system,  its  variance  or  skewness,  etc.)  cannot  be  determined.  Uncertainty 
is  an  abstract  phenomenon,  entirely  subjective  and  peculiar  to  each  indivi¬ 
dual  evaluator.  It  is  this  aspect  of  subjec*ivity  that  gets  us  into  difficulty  in 


—  What  may  be  risk  in  one  case;  e.g.  ,  to  the  military  service  in  terms  of 
the  outcome  for  several  systems,  may  be  uncertainty  in  another.  For 
that  matter,  risk  and  uncertainty  ar»-  not  always  "black  and  white." 


cost-effectiveness  analysis.  (There  are  few  things  more  difficult,  more 
necessary  and  yet  so  ill -defined  as  the  process  of  establishing  specific  per¬ 
formance  requirements,  given  the  mission  definition. 

Note  the  contrast  with  risk.  When  we  refer  to  the  failure  rate  of  a 
component  or  system,  we  may  estimate  the  risk  of  failure  either  through 
past  experience  or  by  comparing  to  similar  components  and  systems.  From 
this  we  may  be  able  to  estimate  what  the  probability  distribution  looks  like. 
Such  a  prediction  of  failure  characteristics  would  then  be  based  on  quantita¬ 
tive  experience  rather  than  opinion. 

Under  uncertainty,  subjective  probabilities  may  be  assigned  through 
the  opinions  and  judgment  of  the  evaluator,  but  there  is  no  method  by  which 
actual  numerical  values  may  be  computed.  Uncertainty  in  this  form  is  not 
readily  insurable;  unlike  risk  it  cannot  be  reduced  directly  to  a  cost.  The 
concept  of  uncertainty  can  be  viewed  very  broadly  to  include  all  circum¬ 
stances  in  which  decisions  must  be  made  without  objective  probabilistic 
knowledge  of  key  events  in  the  future.-^-^  (By  key  event  we  mean  those 
which,  if  we  had  known  about  them  before  they  occurred,  would  have  in¬ 
fluenced  our  decisions  on  resource  use. ) 

C.  USE  OF  RISK  AND  UNCERTAINTY 

Under  risk,  factors  generally  are  found  to  be  technical  or  engineering 
in  nature,  granted  that  some  fali  in  a  gray  area  of  ?(;her  recently  being  of 
an  uncertain  nature  or  are  put  into  a  risk-taking  form  by  assumption.  The 
chemistry  and  physics  of  component  parts,  their  design,  quality  control, 
and  assembly  processes  all  load  to  empirical  probability  estimates  of  per¬ 
formance.  These  parts  in  tu rn-- either  in  terms  of  performance  character¬ 
istics  or  such  measures  as  system  availability--may  be  described  by  the 
parameter  estimates  of  the  resulting  distribution.  Here  risk  becomes  part 
of  the  cost-effectiveness  picture  and  a  design  factor;  expected  system  values 


^  All  "assumptions"  fall  within  the  definition  of  uncertainty,  although 
tome  may  be  subject  to  calculations  of  risk. 
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with  a  narrow  range  or  variance,  or  expected  system  values  in  distributions 
that  are  skewed  to  the  right  (that  is,  there  is  a  smaller  chance  of  lower 
values  occurring)  are  considerations  in  effectiveness  as  well  as  cost.  Alter¬ 
native  configurations  and  redundancies  can  result  in  more  desirable 
distributions  of  system  availability  at  some  finite  cost.  It  is  in  this  sense 
that  risk  is  a  necessary  ingredient  in  the  cost-effectiveness  picture. 

Now  let  us  view  uncertainty  over  and  above  risk.  We  divide  uncertainty 
into  two  areas:  (1)  those  non-engineering  areas  outside  the  system  (the 
mission  and  policy  areas)  and  (2)  those  involved  in  the  system  design, 
operation  and  maintenance.  The  following  examples  are  classified 
accordingly: 

1.  Non -Engineering  --  Extrinsic 

a.  Mission  level  -  peace  versus  war,  limited  versus  central  war, 
political  back-up,  enemy  response,  alternative  force -structure  require¬ 
ments. 

b.  Performance  characteristics  -  relative  to  the  mission,  what 
accuracy,  range,  speed,  vulnerability  will  we  seek?  Are  there  other 
criteria?  What  priority  do  they  have? 

c.  Political  and  economic  environment  -  national  and  international. 

d.  Physical  environment  -  operational  versus  controlled. 

e.  Maintenance  policy  -  number  of  echelons,  repair -discard  level, 
premium  transportation,  technician  capability. 

2.  Intrinsic 

a.  Technical  and  technological  breakthrough  -  e  g.,  the  micro 

area. 

b.  Reliability  -  physics  of  failure. 

c.  Design  vyrsus  maintenance  talent  -  technician  interface  with 


design,  module  sire,  repair -discard,  degree  of  automatic  checkout. 


d.  Cost  functions  -  reliability,  maintainability,  research  and 
development,  lifetime  support  costs. 

To  a  large  extent  uncertainty  is  a  problem  of  imperfect  knowledge, 
with  varying  degrees  of  unknowns.  This  is  the  reason  that  one  must  set 
aside  a  separate  cost  for  uncertainty,  even  though  it  cannot  be  done  with 
the  same  assurance  as  in  cases  of  risk.  Imperfect  foresight,  unless  it  is 
recognised  beforehand,  can  result  in  costly  solutions  for  problems  that 
either  do  not  exist  cr  are  initially  overstated.-^/  Perhaps  even  more 
important,  solutions  which  turn  out  to  be  inadequate  due  to  lack  of  consid¬ 
eration  of  uncertainty,  can  result  in  the  waste  of  resources  that  could  have 
been  available  for  more  productive  purposes.  This  is  particularly  the  case 
on  the  force -structure  level  of  analysis  although  it  holds  also  for  lower 
level  problems  such  as  the  engineering  alternatives  encountered  within  a 
single  system. 

On  a  system  level,  the  degree  to  which  a  Request  for  Proposal  provides 
force- structure  detail  to  the  respondents  can  affect  in  large  part  what  is 
proposed,  since  cost-effectiveness  analyses  by  the  contractor  are  likely  to 
be  sub-optimal  on  a  force  level.  Again,  lack  of  knowledge  (uncertainty) 
restricts  optimisation.  As  a  case  in  point,  perfect  response  to  an  RFP  can 
also  fail  to  take  into  account  the  need  for  alternative  capabilities  over  time 
and  therefore  not  provide  a  sufficient  solution  in  the  total  force  context. 

This  is  typical  of  the  case  of  highly  specialised  and  less  costly  systems 
versus  higher  cost  systems  with  broader  capabilities,  which  over  a  period 
of  time  can  prove  to  be  more  effective. 

Although  the  above  pertains  mostly  to  the  buyer  side  of  the  picture, 
these  are  considerations  for  designers  as  well.-^/  The  designer  must  take 


— /  Imperfect  foresight  always  exists.  The  problem  is  to  account  for  un¬ 
certainty  in  a  manner  that  provides  for  a  level  of  effectiveness  over  a  period 
of  time  with  costs  appropriately  weighed  against  the  likelihood  of  events. 

^  While  not  discussed,  we  also  recognize  the  buyer's  major  uncertainty 
as  to  the  capability  of  the  successful  bid,?-r  to  produce  the  proposed  system 
within  time  and  dollar  limits. 


into  account  the  added  cost*  of  uncertainty  due  to  policies  of  the  Air  Force 
in  addition  to  his  own  irtrinsic  engineering  uncertainties  noted  in  part  above. 

What  does  this  all  add  up  to  in  terms  of  the  purchase  or  design  of  a 
system?  So  overwhelming  are  the  cost  considerations  of  uncertainty,  that 
a  separate  area  in  an  RFP  or  proposal  might  well  be  set  aside  for  it.  Each 
of  the  uncertainties  involved  would  be  delineated  for  a  system;  a  cost  or  set 
of  alternative  costs  (related  if  at  all  possible  to  each  uncertainty  element) 
would  be  generated.  These  costs  may  then  be  evaluated  in  the  final  cost- 
effectiveness  stage.  As  the  system  evolves  in  the  several  gyrations  and 
iterations  during  R  fc  D,  decisions  to  accept  or  reject  the  uncertainty  cost 
may  also  be  made.  The  normal  result  of  this  procedure  would  be  the  con¬ 
ventional  cost-effectiveness  analysis,  with  separate  categories  for  the  cost 
of  risk  and  the  cost  of  uncertainties  as  constraints  which  limit  the  choices 
available  to  key  decision  makers. 

D.  EXAMPLES  OF  RISK  AND  UNCERTAINTY  CONSIDERATIONS 

We  now  delineate  elements  contributing  to  estimates  of  cost  and 
effectiveness,  and  attempt  to  demonstrate  how  changes  in  these  elements 
(or  more  careful  consideration  of  the  resulting  effect  of  them)  may  be  costed 
out  in  a.'.cord  with  previously  discussed  concepts  of  risk  and  uncertainty. 

We  will  show  that  the  analysis  of  risk  and  uncertainty  is  concerned  with 
determining  a  standard  or  base  point  of  reference,  and  then  asking  what 
would  happen  "if". ..  --that  is,  leaving  it  open  to  the  "decision-makers"  as 
to  the  level  of  risk  and  the  amount  of  uncertainty  they  can  permit,  given  the 
likely  costs. 

1.  Cost  of  Risk 

For  a  specified  level  of  over-nil  system  availability,  the  optimised 
(for  cost  versus  reliability  and  maintainability)  co  t  of  the  components  de¬ 
termines  uniquely  the  optimum  configuration  of  the  system,  as  well  as  the 
maintainability  and  reliability  of  the  over -all  system  and  each  of  its 
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component  parts.-^^  This  generalized  statement  may  be  viewed  within  the 
context  of  four  system  characteristics  which  determine  system  effectiveness 
under  the  following  conditions: 

a.  The  mission  time  period  is  defined  and,  after  a  mission 
completion,  facilities  are  available  for  maintenance; 

h.  Units  of  the  system  are  either  failed  or  non -failed  in  terms  of 
the  design  objective; 

c.  A  system  condition  depends  on  the  present  and  future  perfor¬ 
mance  and  not  on  the  past; 

d.  Units  of  the  system  are  mutually  independent. 

Then,  the  four  system  characteristics  are: 

(1)  System  Operational  Availability  -  a  function  of  previous 
operational  reliability,  maintenance  diagnosis,  repair  capability,  logistics, 
etc. 

(2)  System  Readiness  -  a  function  of  diagnosis  during  alert, 
operational  policy,  system  flexibility  and  backup. 

(3)  System  In- Flight  Reliability  (or  Availability)  -  a  function  of 
reliability  and  in-flight  repair  capability. 

(4)  System  Design  Adequacy  -  a  function  of  design  specifications, 
mission  requirements,  performance  capabilities,  external  environment. 

The  fourth  characteristic,  the  most  difficult  and  one  about  which  the 
least  is  known,  ties  in  closely  with  several  points  raised  previously  under 
uncertainty.  The  remaining  characteristics --quantitative  in  nature --a re 
discussed  under  effectiveness  and  may  be  classified  under  risk. 

Examples  of  risk  variables  encountered  in  treating  aspects  of  (1)  to  (3) 
above  are  listed  in  Table  V.  The  variables  of  Table  V  {column  2)  make  up 
the  cost  basis  for  risk  (since  the  variables  and  their  effect  are  quantifiable) 


^  For  proof,  see  Goldman,  A.  S. ,  Economics  of  the  Trade -off  Among 
Reliability.  Maintainability  and  SuddIv.  RM  62-TMP-42.  TEMPO/CE. 
Santa  Barbara,  1962. 
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TABLE  V.  RISK  VARIABLES 
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and  may  be  used  for  evaluating  the  result  of  changes  on  cost  and 
effectiveness  (availability  in  this  case  since  the  variables  refer  to  reliability 
and  maintainability).  The  added  (or  reduced)  costs  resulting  from  changes 
in  variables  of  column  2  result  in  an  effect  on  the  characteristic's  falling 
under  column  4 --in  terms  of  expected  values  and/or  the  effect  on  the  vari¬ 
ances.  It  is  this  added  cost  (from  an  existing  or  reference  standard  point) 
to  attain  the  level  of  confidence  in  mission  success  that  we  classify  under 
the  cost  of  risk. 

2.  Cost  of  Uncertainty 

The  areas  of  uncertainty  depending  in  large  part  on  judgment  are 
as  follows: 

a.  Mission  Requirements  -  These  are  the  ’reason  for  being’  of  the 
system  and  are  usually  in  terms  of  performance  criteria,  represented  by  a 
time  series  of  demands.  Weapons  are  capable  of  multiple  objectives --primary, 
secondary,  etc.  For  each  objective,  more  than  one  criterion  may  apply. 

From  a  cost-effectiveness  standpoint,  it  is  essential  that  Requests 
for  Proposal  provide  not  only  specific  criteria  but  also  alternative  perform¬ 
ance  levels  ranked  according  to  priority.  With  relatively  minor  increases 
in  cost,  it  is  sometimes  possible  to  design  a  more  flexible  weapon  to  fit 
better  into  the  weapons  mix  (rather  than  be  forced  into  multiple  uses  in  a 
later  stage  of  its  life).  In  fact,  the  multiple  capability  of  a  weapon  could 
well  be  the  deciding  factor  in  final  selection  process  based  on  cost-effec¬ 
tiveness  analysis.- —  Ideally,  mission  analysis  should  include  the  game 

^  A  mission  requirement  refers  to  performance  capability  through  time. 
The  performance  capability  might  include  as  factors:  accuracy,  range, 
speed,  and  altitude  (in  general,  engineering  specifications).  These  factors 
are  directly  responsible  for  effectiveness  elements  of  the  weapon--for 
example,  vulnerability  or  penetrability  . 

^  The  area  of  mission  analysis  is  jealously  guarded  by  military  services. 

A  contradiction  seems  to  exist  here  in  that  during  program  definition,  know¬ 
ledge  of  the  mission  by  the  contractor  can  only  aid  in  system  selection. 
Although  there  are  some  striking  exceptions,  the  evidence  points  to  too 
little  contractor  thought  on  the  mission  level.  Such  knowledge  on  the  part  of 
the  contractor  does  not  limit  the  military  in  their  prerogative  of  final  deci¬ 
sion  and  choice. 
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aspect  of  the  opponent's  moves;  this  could  be  done  by  the  military  in 
partnership  with  industry  to  include  "what-if"  moves  by  the  enemy  which 
may  render  a  system  obsolete  or  alter  priorities  in  the  force  mix. 


b.  Environment  -This  refers  to  the  physical,  atmospheric,  and 
climatalogical  environment  in  which  the  system  would  operate  and  is  more 
related  to  technological  capability  than  strategic.,  The  military  must  pro¬ 
vide  its  best  understanding  of  the  unique  aspect  of  environmental  conditions 
and  the  proposal  effort  should  separately  cost  out  a  detailed  plan  of  the  en¬ 
vironmental  work  to  be  undertaken.  Closely  related  here  is  the  likelihood 
of  technological  breakthroughs  and  the  possible  benefits  that  may  result. 


c.  Maintenance  Policy  •  What  support  echelons  are  to  be  included? 
At  what  echelon  should  repair  occur?  What  degree  of  repair?  What  limits 
are  placed  on  transportation?  Inspection  policy?  The  ground  rule?  in  this 
area  must  be  spelled  out  in  the  RFP  since  a  system  "optimised"  for  one 
support  concept  may  not  necessarily  be  optimal  for  another. 

d.  Cost  Area  -This  refers  to  the  elements  of  cost, and  submodels 
going  into  the  total  cost  function. 


The  variability  of  cost  estimates  is  one  thing  that  is  not  likely 
to  be  overcome  in  the  near  future,  but  it  would  help  if  buyers  and  sellers 
were  in  agreement  on  basic  method  and  technique.  Also,  costs  may  be  re¬ 
duced  by  omission  (legally  OK)  thereby  permitting  "slic.it"  cost  proposals  to 
be  the  objective.  To  limit  this  within  reason,  a  detailed  breakdown  of  cost 
areas  and  the  basis  of  the  costing  technique  is  to  be  recommended  covering 


development 

production 

expected  modification  ^ 
operation  I 


time  oriented  with 
cost  elements 


support 


Depending  on  the  weapon  and  stage  of  development,  -  ertain  of  the  above 
phases  are  of  no  concern  to  the  Air  Force.  As  a  general  rule,  only  those 


costa  are  of  importance  v'hich  affect  the  purchase,  operation  and  support 
and  alter  the  cost-effectiveness  choice. 

E.  aiMMA&X 

Each  of  the  cost  areas  may  be  classified  into  the  two  areas  of  risk  and 
uncertainty: 

1.  Risk:  Specific  questions  asked  on  a  "what  if"  basis  to  evaluate 
possible  cost  increases  in  areas  where  increased  effectiveness  can  be 
obtained. 

2.  Uncertainty:  Specific  cost  evaluations  made  where  human  judgment 
is  pa  remount,  and  cost  alternatives  are  significant. 

Requests  for  Proposal  should  distinguish  between  the  two  in  a  manner 
that  forces  a  response  that  has  added  realism  for  cost-effectiveness  evalu¬ 
ation  among  alternatives. 
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SECTION 

EVALUATION  OF  ALTERNATIVES  AND  OPTIMIZATION 

(TASK  9) 

The  purpose  of  this  section  is  to  describe  methods  used  to  evaluate 
alternatives,  rank  them,  and  determine  the  optimum  point  for  the  system 
under  evaluation.  K  is  recognised  that  the  approach  described  is  only  one 
of  many  which  could  be  used,  and  further,  that  it  may  not  be  applicable  in 
all  cases.  The  optimisation  approach  described  involves  adding  cost- 
effectiveness  improvement  items  to  a  basic,  feasible  system.  Alternative 
improvements  are  then  incorporated  in  an  order  such  that  optimal  use  of  As 
affected  resources  is  obtained.  Functional  steps  involved  in  this  portion  of 
the  cost-effectiveness  study  are: 

(1)  Evaluation  of  individual  improvement  alternatives. 

(2)  Ranking  of  individual  alternatives  in  order  of  preference  or 
priority. 

(3)  Determination  of  optimum  or  cut-off  point. 

(4)  Derivation  of  outputs  for  decision  support. 

A.  EVALUATION  OF  INDIVIDUAL  IMPROVEMENT  ALTERNATIVES 

Prior  to  this  point  in  the  over -all  optimisation  process,  alternatives 
have  been  synthesised  and  variables  and  resources  defined  as  stated  in 
Section  HI,  and  the  basic  model  selected,  as  indicated  in  Section  IV.  This 
evaluation  model,  in  conjunction  with  its  related  criteria,  forms  the  final 
basis  for  decision  as  to  optimum  point  and  also,  will  direct  the  basic 
priority  by  which  alternatives  are  ranked.  In  evaluating  individual  alter¬ 
natives,  variables  and  resources  involved  should  be  expressed  in  relation¬ 
ship  to  each  other,  or  as  parametric  relationships  in  such  a  manner  that  they 
match  the  established  model  framework.  Cost-effectiveness  optimisation 
inherently  involves  only  two  major  parameters;  that  of  cost  and  that  of  effec¬ 
tiveness.  Accordingly,  the  models  in  the  individual  evaluations  should  relate 
all  variables  and  resources  to  the  terms  of  these  two  parameters,  or  of  cost 

— ^The  discussion  given  here  is  illustrated  by  example  in  Appendix  II. 


or  value  if  a  profit  form  model  is  used.  Further,  the  format  in  which  these 
relationships  are  expressed  should  be  such  that  it  is  compatible  with  all  of 
the  alternatives  to  be  evaluated  and  ranked  for  the  total  system. 

At  the  more  detailed  levels  of  evaluation,  two  general  situations  can 
confront  the  evaluator.  The  simplest  situation  occurs  when  the  major*'  /  of 
requirements  have  already  been  fixed  by  prior  optimisations  and/or  decisions. 
As  an  example,  let  us  consider  an  electrical  power  system  for  which  the 
power  output  level  and  mission  length  have  been  previously  established. 

The  only  alternatives  then  involve  basic  system  design  approach,  improve¬ 
ment  approaches,  etc.  In  this  case,  individual  alternatives  are  evaluated 
independently.  Estimated  weights,  costs,  reliability  levels,  and  other 
variable  items  are  assessed  as  to  their  impact  on  the  variables  and  re¬ 
sources  appearing  in  the  evaluation  model  structure.  The  other  situation 
that  may  confront  the  evaluator  is  one  in  which  performance  requirements 
»r?  not  fixed  but  are  to  be  optimised  with  the  rest  of  the  system.  In  the 
case  of  the  power  system,  this  may  involve  the  amount  of  power  required, 
mission  lengths,  and  number  of  units  to  be  procured,  in  addition  to  the 
alternatives  noted  in  the  simplest  situation.  In  this  situation,  parametric 
expression  of  variables  is  required.  As  might  be  expected,  many  situations 
will  exist  in  which  the  vise  of  the  matrix  or  the  sise  of  the  family  of  para¬ 
metric  curves  could  become  prohibitive.  In  these  cases,  it  is  generally 
best  to  treat  one  or  more  of  the  variables  as  a  fixed  quantity,  initially,  and 
successively  reiterate  the  quantity  as  the  system  becomes  more  completely 
'  ,-fsrted. 

B.  RANKING  OF  ALTERNATIVES 

The  basic  principle  involved  in  the  ranking  of  improvement  alter¬ 
natives  is  to  incorporate  these  alternatives  in  the  order  which  provides  the 
greatest  amount  of  cost-effectiveness  improvement  per  unit  of  critical  re¬ 
source  expenditure.  Referring  to  some  of  the  basic  models  described  in 
Section  IV,  this  principle  would  apply  as  follows: 

1.  In  a  situation  where  effectiveness  is  to  be  maximised  at  constant 
over-all  cost,  alternatives  would  be  ranked  in  that  order  which  gives  the 
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greatest  amount  of  improvement  in  effectiveness  per  unit  of  critical 
resource  expenditure. 

2.  Against  criteria  requiring  minimisation  of  cost  at  constant 
effectiveness,  alternatives  would  be  ranked  in  the  order  which  provides 
greatest  over  -all  cost  savings  per  unit  of  critical  resource  expenditure. 

3.  For  a  profit  model  criteria,  alternatives  would  be  ranked  in  the 
order  providing  the  greatest  incremental  profit  per  unit  of  critical  resource 
expenditure. 

The  term  "critical  resource"  refers  to  that  program  resource 
which  is  in  most  short  supply  or  most  limiting  with  respect  to  satisfaction 
of  the  optimisation  criteria.  In  the  initial  phases  of  the  analysis,  a  rough, 
approximate  resource  analysis  should  be  accomplished  to  determine  which 
of  the  resources  will  be  the  most  critical.  As  an  alternative,  an  assump¬ 
tion  can  be  made  as  to  which  of  the  resources  is  critical.  The  criticality 
of  the  resource  is  verified  after  initial  determination  of  the  optimum  point. 

If  the  preliminary  estimate  of  resource  criticality  is  in  error,  the  ranking 
of  alternatives  must  be  rerun  in  accordance  with  whatever  resource  has  in 
fact,  proven  to  be  most  critical. 

In  applying  this  technique  to  the  design  of  a  hardware  system,  a 
number  of  constraints  affecting  the  ranking  of  alternatives  will  become 
apparent.  These  constraints  are  the  result  of  technical  or  hardware  inter¬ 
faces  and  are  of  two  basic  types: 

Type  I  -  An  alternative  cannot  be  incorporated  until  after  some  other 
alternative  has  been  incorporated. 

Type  II  -An  rlUrrative  cannot  be  incorporated  if  some  other  alter¬ 
native  has  been  previously  incorporated. 

An  additional  factor  influencing  the  ranking  is  caused  by  non-linear¬ 
ities  in  effectiveness  and  resource  elements.  These  non-linearities  are 
such  that  the  magnitude  of  effectiveness,  cost,  or  profit  improvement  as¬ 
sociated  with  a  given  alternative  will  depend  upon  the  point  in  the  ranking  at 
which  the  alternative  is  installed  in  the  system. 
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These  non-linearities,  in  conjunction  with  the  above  constraints, 
lead  to  the  following  observations : 

a.  Ranking  is  an  iterative  process  wherein  installation  of  the  best 
alternative  forces  re-evaluation  of  all  remaining  alternatives  prior  to  in¬ 
stallation  of  the  next  best  alternatives,  and  so  on. 

b.  Occasionally,  at  low  levels  of  reliability,  a  reversal  in  ranking 
factor  will  occur  which  cannot  be  eliminated. 

c.  Some  alternatives  will  not  expend  any  of  the  critical  resource 
and  thus  will  have  a  ranking  factor  of  ±  oo .  Those  involving  net  loss,  (-ao ), 
may  change  to  +  oo  at  some  point  in  the  ranking  process. 

d.  An  alternative  subject  to  Type  I  constraint  above  may  be  ranked 
earlier  if  combined  with  its  constraining  alternative. 

e.  An  alternative  subject  to  Type  II  constraint  above  may  eventually 
be  of  such  value  that  it  is  installed  at  the  expense  of  deleting  its  constrain¬ 
ing  alternative. 

C.  DETERMINATION  OF  OPTIMUM  POINT 

The  purpose  of  this  step  is  to  establish  the  over-all  maximum,  mini¬ 
mum,  or  cut-off  point  against  the  established  optimisation  criteria.  The 
use  of  the  term  "optimum"  must  be  clarified  by  saying  that  a  mathematical 
or  theoretically  rigorous  definition  of  the  term  as  applied  here  is  not 
intended.  As  should  be  obvious,  the  real  world  as  well  as  the  analysis  and 
optimisation  technique  used  will  impose  constraints  and  limitations  on  the 
ability  to,  in  fact,  arrive  at  an  optimum  point  or  even  to  know  with  certainty 
that  a  point  thought  to  be  optimum  is  in  fact  correct.  Further,  in  the  tech¬ 
nique  shown  here,  the  so-called  optimum  point  is  determined  by  an  iterative 
or  feed-back  process,  in  which  an  initial  apparent  or  pseudo -optimum  is 
first  determined  and  the  process  is  reiterated  a  number  of  times  to  arrive 
at  a  closer  approximation  of  the  true  optimum  point. 

The  optimisation  technique  described  here  will  take  somewhat  dif¬ 
ferent  forms  depending  upon  the  circumstance.  Accordingly,  this  technique 
will  be  dismissed  in  the  light  of  the  several  cases  which  can  occur.  The 
first  case  which  we  will  discuss  involves  maximization  of  effectiveness. 
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The  result*  of  the  ranking  process  are  shown  in  curve  form,  Figure  14.  In 
this  case,  expenditure  of  the  critical  resource  does  not  reduce  effectiveness 
and  the  optimum  point  is  a  cut-off  which  occurs  at  the  limit  of  the  available 
resource.  Figure  15  shows  a  different  situation  involving  the  same  condi¬ 
tions  as  the  one  above,  except  that  effectiveness  reaches  a  maximum  and 
proceeds  to  decrease.  If  the  limit  of  the  critical  resource  is  beyond  the 
maximum  point  then  tne  maximum  becomes  th«  optimum. 

The  second  basic  case  which  cart  occur  involves  a  situation  where 
expenditure  of  the  critical  resource  diminishes  effectiveness.  If  the  de¬ 
crease  in  effectiveness  is  linear  with  respect  to  the  resource  expenditure, 
then  the  impact  of  the  resource  expenditure  can  be  accounted  for  in  the 
initial  ranking.  The  approach  used  to  determine  the  optimum  point  will 
then  take  the  same  form  as  in  the  cases  described  above.  However,  if  the 
relationship  of  effectiveness  to  critical  resource  expenditure  is  non-linear, 
then  its  impact  cannot  conveniently  be  handled  during  the  ranking  process. 

In  this  case,  its  impact  is  ignored  during  the  ranking  and  a  curve  is  plotted 
based  on  gross  effectiveness  improvement  per  unit  of  critical  resource 
expenditure.  A  second  curve  defining  the  relationship  of  effectiveness  to 
critical  resource  expenditure  is  shown  in  Figure  16.  The  optimum  point  is 
determined  by  superimposing  these  two  curves  as  shown  in  Figure  17.  The 
optimum  point  occurs  at  the  point  of  tangency  of  the  two  curves. 

The  same  general  forms  and  logic  used  in  the  cases  illustrated  above 
applies  to  use  of  the  profit  model  or  to  cases  where  effectiveness  is  main¬ 
tained  as  a  constant  and  cost  is  being  minimised. 

As  was  noted  above,  the  initial  determination  of  optimum  point  is 
made  on  the  assumption  that  one  particular  resource  is  critical.  In  some 
cases,  the  necessity  of  selecting  a  critical  regource  is  eliminated  or  the 
number  of  resources  from  which  the  critical  item  must  be  selected  is  re¬ 
duced  if  it  is  possible  to  relate  one  resource  in  terms  of  another.  In  the 
optimisation  techniques  used  here,  this  is  generally  feasible  only  in  cases 
where  most  of  the  relationships  are  linear  or  nearly  so.  In  the  usual  case 
there  will  still  be  two  or  more  resources  involved  in  the  determination  of 
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Critical  Resource  Expended 


FIGURE  14 

MAXIMIZATION  OF  EFFECTIVENESS  -  OPTIMUM  POINT 
AT  THE  LIMIT  OF  THE  CRITICAL  RESOURCE 


Critical  i\e source  Expended 


FICURE  15 

MAXIMIZATION  OF  EFFECTIVENESS  -  LIMIT  OF  THE  CRITICAL 
RESOURCE  BEYOND  THE  OPTIMUM  POINT 
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Critical  Resource  Expended 


FIGURE  16 

RELATIONSHIP  OF  EFFECTIVENESS  TO  CRITICAL 
RESOURCE  EXPENDITURE 


FIGURE  J7 


LOCATING  OPTIMUM  POINT  WHEN  RELATIONSHIP  OF 
EFFECTIVENESS  TO  CRITICAL  RESOURCE  EXPENDITURE  IS 

NON-LINEAR 


the  optimum  point.  In  such  a  case,  when  the  optimum  point  has  been 
determined,  the  remaining  or  non-critical  resources  must  then  be  surveyed 
to  determine  whether  or  not  their  availability  has  been  completely  expended. 
If  the  availability  of  a  resource  previously  thought  to  be  non-critical  has 
been  exceeded,  then  the  alternatives  must  be  re -ranked  in  order  to  conserve 
this  resource  rather  than  the  one  previously  thought  to  be  critical,  and  the 
optimum  point  redetermined. 

In  general,  at  this  stage  of  analysis,  one  or  more  of  the  non-critical 
resources  will  be  in  plentiful  supply  at  the  point  where  the  critical  resource 
has  been  exhausted,  or  is  limiting.  Generally,  in  these  cases,  further  im¬ 
provement  can  still  be  obtained  by  use  of  alternatives  which  tend  to  deplete 
the  plentiful  resources  in  exchange  for  increasing  the  supply  of  the  one 
which  is  critical.  Synthesis  and  evaluation  of  alternatives  of  this  nature 
are  best  accomplished  after  the  first  step  ranking  and  initial  optimisation 
point  is  determined. 

D.  MODEL  OUTPUTS  FOR  DECISION  SUPPORT 

Unless  the  outputs  or  final  results  of  the  cost  optimization  study  is 
in  or  can  be  placed  in  a  form  suitable  for  support  of  program  decisions, 
then  the  application  of  such  optimisation  principles  becomes  only  an  aca¬ 
demic  exercise.  The  type  of  decision  support  outputs  which  can  be 
generally  derived  from  the  optimization  technique  shown  here  can  be 
categorized  as  follows: 

(1)  Definition  of  design  and  mission  associated  with  the  optimum 
point. 

(2)  Criteria  for  evaluation  and  decision  on  future  improvement 
alternatives. 

(1)  Parametric  data  for  use  in  studies  of  other  systems. 

The  basic  optimization  tcchniq  's  described  here  involve  incorpora¬ 
tion  of  alternative  approaches  up  to  the  optimum  or  cut-off  point.  Inherently, 
this  approach,  once  the  optimum  or  cut-off  point  has  been  defined  is  such 
that  a  definition  of  the  configuration,  mission  and  other  scenario  associated 
\iuth  the  optimum  point  is  readily  determined  by  reading  out  the  definition  of 


those  alternatives  which  were  incorporated  prior  to  reaching  the  optimum 
point.  As  the  program  progresses  through  Definition  Phase  and  through  its 
operational  life,  the  description  of  alternatives  involved  in  the  optimisation 
process  will  become  increasingly  definitive  and,  accordingly,  the  definition 
of  the  system  corresponding  to  the  optimum  point  will  become  increasingly 
definitive.  In  addition  to  system  definition  outputs  associated  with  the 
optimum  points,  a  number  of  other  requirements  type  parameters  may  be 
calculated.  Examples  of  such  parameters  are  reliability  level,  maintain¬ 
ability  level,  required  number  of  operational  units,  etc.  In  short,  it  should 
be  possible  to  either  take  directly  from  the  plots  from  which  the  optimum 
point  was  derived  or  to  calculate,  based  on  the  resulting  system  definition, 
quantitative  values  for  all  of  the  principal  elements  of  effectiveness  and 
cost. 

It  must  be  recognized  that  in  the  time  period  following  completion  of 
an  optimization  analysis,  further  system  improvement  alternatives  will  be 
considered  for  incorporation.  The  scope  and  complexity  of  an  over-all  sys¬ 
tem  optimization  analysis  are  generally  such  that  it  is  undesirable  to  rerun 
the  entire  study  each  time  such  an  improvement  alternative  is  to  be  con¬ 
sidered.  As  a  result  then,  it  is  generally  desirable  to  derive  evaluation 
models  and  criteria  from  the  optimum  point  and  the  resource  relationships 
accompanying  this  point,  in  such  form  that  future  improvement  alternatives 
can  be  evaluated  somewhat  independently  of  the  over-ail  system  analysis. 
There  is  admittedly  an  implicit  danger  in  following  such  an  approach  in 
that,  when  a  large  enough  number  of  such  alternatives  have  been  incorpor¬ 
ated  into  the  system,  the  original  optimization  analysis  from  which  the 
individual  evaluation  criteria  was  derived  becomes  invalid,  thus  admitting 
the  possiblity  of  erroneous  decisions.  Determination  of  the  circumstances 
and/or  frequency  under  which  the  total  optimization  must  be  reiterated  is  a 
matter  of  individual  judgment.  General  guidelines  for  formulation  of 
evaluation  criteria  for  future  improvements  are; 

a.  It  is  desirable  to  issue  such  criteria  a  form  such  that  an 
individual  designer  can  use  it. 


b.  The  form  mu<t  be  euch  that  a  aingle  criteria  or  cut-off  ia  uaed. 
Thua,  aupporting  data  and  relationahipa  muat  be  provided  ao  that  all  vari- 
ablea  and  reaourcea  which  are  involved  in  future  evaluationa  can  be  related 
in  terma  of  common  denomination,  namely,  the  denomination  aaaociated 
with  the  aelected  criteria. 

The  third  type  of  deciaion  aupport  output  which  can  be  gained  from 
an  optimisation  analyaia  ia  baaically  the  type  of  parametric  data  uaed 
aa  input  a  to  future  atudiea  conducted  at  a  more  groaa  level  on  aimilar 
ayatema. 


SECTION  IX 


INTERPRETATION  OF  RESULTS. 

SENSITIVITY,  AND  LIMITATIONS 
(TASKS  10.  11) 

The  cost-effectiveness  indices  derived  from  a  given  set  of  input 
data  used  in  studies  and  models  of  the  type  described  in  this  report  should 
be  considered  as  measures  of  "goodness"  or  "badness"  of  a  particular 
system  or  system  configuration.  The  validity  of  the  indice*  then,  depends 
on  the  validity  of  the  model  and  the  input  data.  But  even  though  a  particular 
submodel  may  represent  reality  only  on  a  gross  basis,  or  a  particular 
piece  of  input  data  may  be  only  a  gross  estimate  ol  reality,  the  resulting 
index  may  still  deserve  confident  consideration  as  a  measure  of  goodness. 

The  key  is  the  sensitivity  of  the  result  to  such  gross  representations.  In 
sensitive  areas  associated  with  risk  or  uncertainty,  "warning  flags"  must 
be  attached  and  some  idea  of  upper  and  lower  bounds  for  the  measure 
should  be  given.  These  are  called  sensitivity  checks. 

Sensitivity  checks  aie  needed  since  the  output  of  a  cost-effectiveness 
optimisation  study  is  used  to  support  program  decision.  Sensitivity  checks 
are  intended  to  determine  the  effect  of  uncertainty  in  the  output  data  on  the 
decisions  involved.  Three  fundamental  types  of  sensitivity  checks  will 
generally  be  applicable  to  the  results  of  most  cost-effectiveness  studies. 

They  are: 

(1)  Sensitivity  to  basic  system  or  mission  requirements, 

(2)  Sensitivity  to  uncertainties  in  estimated  or  extrapolated  data, 

(3)  Validity  of  simplifying  assumptions  or  arbitrarily  fixed  variables. 

The  basic  steps  involved  in  a  sensitivity  analysis  are: 

(1)  An  estimate  or  guess  should  be  made  as  to  the  possible  numeri¬ 
cal  range  of  uncertainty  involved.  Per  the  deiinition  of  uncertainty  outlined 
in  the  previous  sections,  it  should  be  recognized  that  such  estimates  or 
guesses  will  generally  be  unsupported  by  any  data  or  background  information. 
They  are  usually  a  matter  of  judgment. 


(Z)  Using  both  the  maximum  and  minimum  values  of  the  range,  the 
optimization  analysis  or  necessary  portions  thereof,  should  be  rerun.  The 
results  of  the  nominal  and  extreme  values  can  then  be  compared  as  required 
to  determine  whether  decisions  which  would  have  been  derived  from  the 
nominal  analysis  would  be  altered  if  the  extreme  values  were  believed. 

(3)  If  it  becomes  apparent  that  the  possible  range  of  uncertainty 
does  have  significant  effect  on  output  decisions,  steps  should  be  taken  to 
reduce  the  range  of  uncertainty,  through  either  improvements  in  input  data 
or  testing  for  more  or  better  data,  improved  analysis  techniques  or  pre¬ 
paration  of  estimates  at  a  lower  level.  In  general,  experience  with  this 
type  of  analysis  has  shown  that  only  a  small  percentage  of  study  inputs 
involving  a  range  of  uncertainty  will  be  such  that  this  range  of  uncertainty 
will  influence  output  decisions. 

As  an  alternative,  there  are  some  situations  where  the  basic 
dejign  of  the  system  can  be  altered  in  such  a  manner  that  the  system  is  no 
longer  sensitive  to  the  estimated  range  of  uncertainties.  If  it  is  not  possible 
to  remove  the  effects  of  the  uncertainty  range  on  the  decisions  involved,  this 
effect  should  be  shown  in  visible  or  parametric  form  along  with  analysis 
results. 

If  they  are  to  be  of  value,  the  results  of  cost-effectiveness  studies 
must  be  given  in  terms  which  are  meaningful  to  those  who  make  decisions 
and  understand  the  implications  and  results  of  these  analyses.  Thus,  the 
analysts  should  appreciate  the  problems  of  communication  with  a  broad 
spectrum  of  people  including  design  engineers,  company  managers,  military 
managers,  military  planners  and,  sometimes,  congressmen  and  the  general 
public . 

In  interpreting  the  results  of  these  studies,  it  must  be  remembered 
that  the  state-of-the-art,  resource  constraints,  political  and  military 
thinking  and  philosophy,  enemy  posture,  etc. ,  are  in  a  constant  state  of 
flux.  Thus,  these  results  should  not  become  associated  with  hard,  fast, 
unchanging  rul?s.  A  current  finding  that  a  reliability  of  0.9  is  best  for  a 
particular  component  should  not  become  permanent  dogma.  The  results 


should  never  be  the  basis  for  hindering  research.  Rather,  they  should 
provide  better  guidelines  to  design  exploratory  experiments  so  that  the 
results  of  the  tests  will  yield  the  more  fruitful  and  a  greater  quantity  of 
information  for  further  analyses. 

The  limitations  of  cost-effectiveness  studies  have  already  been 
suggested  in  the  foregoing  paragraphs.  The  reader  should  bear  in  mind 
that,  whatever  shortcomings  or  dangers  may  be  associated  with  analytical 
studies  such  as  these,  decisions  based  on  intuition,  experience  which  has 
not  been  thoroughly  analysed,  or  a  sample  of  personal  opinions  ("gut" 
feelings)  are  certainly  less  defensible  and  more  subject  to  omission  of 
important  factors.  One  would  not  build  a  bridge  by  intuitive  design,  over¬ 
looking  sound  structural  engineering  practice.  Yet,  many  unknowns  exist 
in  regard  to  material  mechanics  and  random  loading  behavior  of  structures. 

Although,  in  a  sense,  statements  on  limitations  of  cost-effectiveness 
analyses  may  be  regarded  a*  platitudes,  we  present  some  of  them  here  as 
reminders. 

1.  Cost-effectiveness  indices  cannot  be  meaningful  unless  derived 
from  a  model  which  represents  the  "real  world"  fairly  closely.  Reality 
should  not  be  buried  under  mountains  of  details  nor  does  great  detail,  by 
itself,  create  reality  in  a  model. 

2.  It  must  be  remembered  that  cost-effectiveness  analysis  is  an 
iterative  process.  Early  results  should  not  be  permitted  to  create  such  a 
lasting  impression  (favorable  or  unfavorable)  as  to  lead  one  to  ignore  the 
results  of  later  refinements.  This  can  lead  to  disillusionment  on  the  part 
of  all  concerned  and,  thence,  lead  to  abandonment  of  a  valuable  tool. 

3.  Cost-effectiveness  analysis  can  never  replace  good  engineering 
and  management  practices.  It  should  be -regarded  as  a  supplementary  tool 
ta  provide  meaningful  information.  Final  decisions  must  still  be  based  upon 
sound  judgment.  This  must  be  particularly  emphasized  since  too  many  poli¬ 
tical,  psychological  (e.g.,  an  individual's  drive  to  solve  a  particular  pro¬ 
blem),  prestige  value,  and  other  factors  cannot  be  considered  in  a  satisfac¬ 
tory  manner  at  this  time  in  such  analyses. 


4.  When  results  are  sensitive  to  factors  associated  with  high 
degrees  of  risk  or  uncertainty,  "warning  signs"  must  be  posted.  The 
results  must  then  be  used  judiciously  in  making  decisions. 

In  much  of  what  has  been  said  in  the  foregoing,  there  is  an  obvious 
attempt  to  build  up  the  importance  of  cost-effectiveness  consciousness. 
Considerable  emphasis  has  been  placed  on  developing  models  for  obtaining 
cost-effectiveness  indices  and  optimisation  thereof.  However,  it  must  be 
remembered  that  these  do  not  provide  a  final  answer.  They  do  provide 
guidelines,  but  Judgment  must  still  play  a  large  part. 

18  / 

Perhaps  this  is  best  expressed  by  Dr.  Alain  Enthoven's  statement:-— 
"Do  judgment  and  experience  have  no  place  in  this  approach  to  choice  of 
weapon  systems  and  strategy  and  design  of  the  defense  programs  ?  Quite 
the  contrary.  The  statement  that  the  issue  is  judgment  versus  computers 
is  a  red  herring.  Ultimately  all  policies  are  made  and  all  weapon  systems 
are  chosen  on  the  basis  of  judgments.  There  is  no  other  way  and  there 
never  will  be.  The  question  is  whether  those  Judgments  have  to  be  made  in 
the  fog  of  inadequate  and  inaccurate  data,  unclear  and  indefinite  issues,  and 
a  welter  of  conflicting  personal  opinions,  or  whether  they  can  be  made  on 
the  basis  of  adequate,  reliable  information,  relevant  experience,  and  clearly 
drawn  issues.  The  point  is  to  render  unto  computers  the  things  that  are 
computers'  and  to  judgment  the  things  that  are  judgment's.  In  the  end,  there 
is  no  question  that  analysis  is  but  an  aid  to  judgment  and  that,  as  in  the  case 
of  God  and  Caesar,  judgment  is  supreme.  " 

Thus,  although  there  are  limitations  in  this  modeling  process  to 
obtain  cost-effectiveness  indices,  it  must  be  remembered  that  this  approach 
allows  us  to: 

1.  Organise  and  set  into  proper  perspective  the  many  alternatives 
of  the  problem. 


From  a  lecture,  "Decision  Theory  and  Systems  Analysis.  "  delivered 
during  the  Distinguished  Lecture  Series,  sponsored  by  the  Board  of  Trade 
Science  Bureau,  Washington,  D.  C.,  Decembers,  1963. 
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Establish  many  "if-then"  statements  pertaining  to  the  alter¬ 
natives  of  the  problem, 

Evaluate  properly  data  uncertainties. 

Examine  many  cases  quickly  which  would  require  years  of 
simulated  combat  to  test, 

Implore  systematically  those  cases  which  cannot  be  tested 
(you  cannot  go  to  war  to  test  system  effectiveness). 


SECTION  X 


CONCLUSIONS  AND  RECOMMENDATIONS 

A.  DISCUSSION  ON  ACCOMPLISHMENTS  AND  IMPLEMENTATION 

The  work  of  Tank  Group  IV  of  the  Weapon  Syetem  Effectiveness  In¬ 
dustry  Advisory  Committee  has  provided  a  framework  for  cost-effectiveness 
studies  and  a  common  basis  for  using  the  results  of  these  studies  in  evalu¬ 
ating  systems.  No  single  "cookbook"  model  or  set  of  rules  is  provided 
since  such  does  not  seem  possible  or  desirable  at  this  point  in  time.  In¬ 
stead.  appropriate  formulae  and  check  lists  of  factors  which  must  be  con¬ 
sidered  for  relevancy  are  provided. 

During  the  course  of  this  study,  existing  deficiencies  and  problem 
areas  have  become  apparent.  Many  of  these  have  been  examined  by  this  and 
other  task  groups  of  WSEIAC.  One  of  the  principal  problems  is  associated 
with  the  fact  that,  although  the  concept  is  not  new,  the  "Cost -Effectiveness 
Art  (or  science)"  is  really  in  its  infancy.  There  is  presently  only  little 
appreciation  for  what  is  involved  in  such  studies  and  only  slightly  more 
appreciation  for  the  usefulness  of  the  results  of  such  studies.  There  is  a 
general  fear  that  an  attempt  is  being  made  to  find  a  model  to  "solve  all 
problems  for  all  time,"  and  that  we  are  "looking  for  black  cats  in  a  dark 
basement.  "  Since  these  are  quotes  from  qualified  people  deeply  involved  in 
various  aspects  of  development  or  evaluation  of  weapon  and  space  systems, 
it  seems  fair  to  say  that  there  is  an  apparent  need  for  indoctrination  and 
clarification  of  intent.  Recognition  of  the  problems  associated  with  estiir*- 
ting  performance  and  cost  for  system  elements  at  early  stages  of  develop¬ 
ment  should  not  lead  to  avoiding  the  issue  of  system  evaluation  at  that  time. 
Instead,  this  recognition  should  point  up  the  need  for  such  evaluations  in 
view  of  the  risks  and  uncertainties  involved.  Admittedly  these  early  evalu¬ 
ations  may  leave  much  to  be  desired  in  terms  of  preciseness  of  the  results; 
but,  at  least,  they  permit  recognition  of  this  imprcciseness  on  a  quantitative 
basis.  Further,  they  should  provide  a  better  basis  for  decision  in 


concentrating  efforts  to  erase  key  uncertainties  before  irrevocably 
committing  large  sums  of  money. 

There  are  already  many  data  available  from  past  and  present  sys¬ 
tems  which  permit  application  of  available  techniques  in  determining  reli¬ 
ability,  maintainability,  penetrability,  survivability,  lethality,  performance 
and  costs.  This  report  reviews  the  technique  and  the  types  of  data  required. 
There  is  still  much  to  be  done  in  improving  and  validating  existing  submodels 
and  in  identifying  and  quantifying  additional  factors  which  should  be  consi¬ 
dered  in  cost-effectiveness  analyses.  There  is  also  necessity  for  inducing 
additional  effort  in  formulating  and  disseminating  meaningful  measures  of 
cost-effectiveness  for  a  broader  spectrum  of  systems. 

The  approach  and  models  given  in  this  report  should  be  recognised 
as  being  useful  to  both  the  military  and  industry  and  provide  a  "check  list" 
and  a  framework  for  performing  cost-effectiveness  studies.  They  should 
be  used.  At  the  very  least,  the  elements  listed  must  be  considered  for 
relevancy.  No  single  model  is  given  for  considering  all  systems,  simply 
because  one  cannot  "model  the  universe. "  However,  the  elements,  models, 
submodels,  and  "warning  signals"  (e.g.,  risks  and  uncertainties)  given 
here  constitute  the  basis  for  what  must  be  done,  considering  costs, 
effectiveness,  and  constraints,  to  derive  figures  of  merit  for  use  in  the 
decision  process. 

The  benefits  of  application  of  cost-effectiveness  studies  are  readily 
apparent.  The  use  of  quantitative  measures  of  cost-effectiveness  and 
decisionary  emphasis  on  them  automatically  induces  an  economic  conscious¬ 
ness.  It  adds  another  dimension  to  system  design,  which  was  previously 
dominated  solely  by  technical  considerations.  With  additional  application  of 
these  tools  and  subsequent  decision  making  based  on  cost-effectiveness 
measures,  that  point  will  be  reached  where  it  should  be  possible  to  specify 
a  quantitative  system  effectiveness  which  can  be  enforced  and  to  establish 
cost-effectiveness  goals.  Ce  tainly  this  will  encourage  contractors  to  pro¬ 
duce  better  systems  for  less  money. 
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The  cost-effectiveness  approach  must  not  be  introduced  as 
"revolutionary,"  or  disillusionment  is  inevitable.  Although  this  approach 
should  take  its  place  as  rapidly  as  possible,  the  place  must  be  appropriate 
a.id  the  process  of  introduction  must  be  "evolutionary.  "  Time  is  necessary 
to  improve  the  approach,  the  models,  the  data  collection  process,  "optimi¬ 
zation"  technique,  and  to  develop  a  better  appreciation  for  the  usefulness  of 
this  management  tool.  It  must  be  remembered  that  good  judgment  by  intelli¬ 
gent  and  qualified  people  is  necessary  in  both  the  cost-effectiveness  studies 
and  in  the  final  decisidn  process. 

B.  CONCLUSIONS 

1.  Cost-effectiveness  as  an  art,  science,  or  discipline  is  still  in 
its  infancy. 

2.  No  single  "cookbook"  method  for  performing  cost-effectiveness 
studies  is  possible  or  desirable  at  this  time. 

3.  Task  Group  IV  has  shown  that  a  general  framework  for  perform¬ 
ing  cost-effectiveness  studies  on  a  more  common  basis,  leading  to  standard¬ 
isation  of  methods  and  evaluation  of  results,  is  possible  and  very  desirable. 

4.  Standardisation  and  availability  of  data  to  perform  effective 
cost-effectiveness  studies  is  seriously  lacking. 

5.  Many  persons  have  little  knowledge  of  or  appreciation  for 
either  the  usefulness  orthe  intent  o.  cost-effectiveness  studies. 

6.  More  study  and  education  of  the  methodology  aud  techniques  for 
performing  cost-effectiveness  studies  is  needed. 

C.  RECOMMENDATIONS 

The  Air  Force  should: 

1.  Continue  its  efforts  to  indoctrinate  their  personnel  in  the 
principles  and  techniques  of  cost-effectiveness  and  instill  in  them  a  cost- 
effectiveness  consciousness  on  an  organisational  and  individual  basis  and 
insist  that  their  prime  contractors  do  the  same.  Relevant  training  programs 
(e.  g. ,  for  maintainability  and  system  program  office  pet  sonnet)  should  con¬ 
tain  appropriate  "doses"  of  cost-effectiveness  information. 
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2.  Provide  guidelines  to  be  used  in  specifying  and  evaluating  cost- 
effectiveness  studies  ir  requests  for  proposals,  contracts,  and  other  pro¬ 
gramming  documents. 

3.  Apply  the  cost-effectiveness  evaluation  approach  on  a  cradle- 
to-grave  basis  with  the  details  included  being  appropriate  to  the  level  of 
information  available. 

4.  Identify  those  elements  in  cost-effectiveness  analysis  on  which 
further  research  must  be  done  to  increase  the  utility  of  this  approach  to 
system  evaluation  and  to  improve  the  tools  for  performing  cost-effective¬ 
ness  analyses.  This  should  include  development  of  better  submodels, 
methods  for  building  up  comprehensive  models,  approaches  to  "optimiza¬ 
tion,  "  and  standardising  cost  analyses. 

5.  Continue  to  support  a  program  to  assure  that  a  data  collection, 
analysis,  storage,  and  retrieval  system  is  appropriate  for  supporting 
cost-effectiveness  analyses.  The  data  system  should  contain  as  many 
functional  relationships  as  possible,  as  well  as  raw  data,  so  that  raw  data 
do  not  have  to  be  analyzed  repeatedly  in  a  similar  way  for  similar  studies. 

6.  Provide  a  better  method  for  specifying  the  over-all  system 
environment  (e.g.,  enemy  systems  and  practices)  to  permit  establishing 
realistic  models  for  the  "scenario"  in  which  both  military  and  contractors 
have  confidence.  The  aim  of  the  method  should  be  to  reduce  the  possibility 
of  eliminating  superior  systems  through  misinterpretation  of  relevant 
environments  or  by  inconsistent  evaluations. 

7.  Exercise  considerable  care  to  assure  that  the  cost-effectiveness 
approach  does  not  get  a  "revolutionary  hold"  on  the  decision-making  pro¬ 
cesses  but  rather  that  it  is  introduced  as  "evolutionary." 

The  above  recommendations  will  necessarily  have  to  be  transformed 
into  policy  documents  (regulations,  program  management  instructions, 
manuals  and  the  like)  and  into  specific  projects  and  Air  Force  management 
objectives.  In  addition,  some  agency  must  be  designated  to  implement 
and/or  monitor  each  recommendation.  Whether  this  should  be  done  through 
normal  administrative  procedures  or  through  a  special  office  must  be  decided. 
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APPENDIX  I 


COMPILATION  OF  ABSTRACTS  FROM 
EXAMPLES  OF  VOLUME  III  OF 
TASK  GROUP  IV  FINAL  REPORT 

EXAMPLE  A  -  AIRCRAFT  SYSTEM  OPTIMIZATION 

A  system  cost-effectiveness  model  is  developed  for  an  Air  Force 

training  base  at  which  daily  bomber  training  flights  are  made.  In  the  event 
of  enemy  attack,  the  base  bomber  force  is  assigned  to  targets.  The  objec¬ 
tive  of  the  example  is  to  illustrate  the  optimisation  of  the  bomber  effective¬ 
ness  by  trading  off  reliability,  maintainability,  performance  and  cost  factors. 
The  system  effectiveness  model  is  developed  along  the  mathematical  lines 
presented  by  Task  Group  II  in  Volume  II  of  their  final  report.  Optimization 
is  accomplished  by  computing  and  comparing  the  costs  of  eight  possible 
measurement  and  support  policies  in  terms  of  two  alternative  figures  of 
merit: 

(1)  For  each  target,  there  will  be  a  0. 95  probability  that  at  least 
one  of  the  attacking  aircraft  will  successfully  accomplish  the 
bombing  run. 

(2)  There  will  be  an  average  success  probability  of  0.  95  for  all 
assigned  targets. 

A  significant  aspect  of  this  example  is  its  illustration  of  the  need 
for  re-evaluating  the  criterion  for  optimization  in  terms  of  the  realized 
output  of  the  evaluation  effort. 

EXAMPLE  B  -  RELIABILITY  ALLOCATION 

A  method  for  allocating  system  reliability  requirements  among 
subsystems  (or  lower  level  units)  is  presented.  The  method  considers 
serial  and  redundant  interconnections  among  the  subsystems.  The 


relationship  between  system  reliability  requirements  and  system 
effectiveness  requirements  is  considered. 

EXAMPLE  C  -  BALLISTIC  MISSILE  PAYLOAD  ALLOCATION 

Each  element  of  a  ballistic  missile's  payload- -warhead,  guidance 
and  penetration  aids — will  increase  in  effectiveness  with  an  increase  of 
weight  allocated  to  the  element.  For  a  missile  that  is  to  be  employed 
against  a  defended  "point"  target,  this  example  presents  a  method  for 
determining  the  optimum  division  of  the  missile's  payload  between  the 
three  competing  (for  weight)  elements,  when  their  individual  weight- 
effectiveness  relationships  are  known.  For  the  case  of  a  single  missile  per 
target,  ut.:ng  a  most  basic  application  of  the  stepwise  optimization  philo¬ 
sophy  of  dynamic  programming,  the  problem  is  formulated  as  a  two-stage 
weight  allocation  process.  The  first  stage  determines  the  optimum  trade¬ 
off  between  warhead  (lethal  radius)  and  guidance  (CEP);  the  second  stage 
determines  the  optimum  division  between  penetration  aids  and  an  optimum 
mix  of  warhead  and  guidance.  The  simple  arithmetical  method  that  results 
is  demonstrated  by  an  example.  The  same  optimisation  process  is  useful 
for  the  cases  of  sequential  and  simultaneous  multiple  missile  employment 
per  target.  Although  this  design  optimisation  problem  can  be  solved,  func¬ 
tionally,  for  the  modes  of  missile  employment  considered,  its  applicability 
to  a  real  allocation  problem  is  confounded  by  the  design,  intelligence  and 
employment  estimates  required  in  the  analysis.  Use  of  this  method  could 
show,  however,  the  influence  of  the  estimated  uncertainties  cn  the  optimal 
payload  division  and  could  thereby  serve  as  a  useful  point  of  departure  for 
design  compromises. 

EXAMPLE  D  -  OPTIMIZING  A  PRELAUNCH  CHECKOUT 

This  example  presents  a  procedure  for  determining  the  optimum 
test  content  of  an  ICBM  prelaunch  checkout  that  is  subject  to  a  time  con¬ 
straint.  Cost  considerations  are  not  introduced  as  a  constraint,  but  instead 
arc  employed  after  the  test  content  has  been  optimised  for  each  possible 
test  duration  constraint  in  order  to  select  between  ti  iignr.  An  example  is 
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given  and  references  are  cited  that  contain  an  explanation  of  the  estimation 
of  the  parameters  associated  with  the  design  technique. 


EXAMPLE  E  -  MISSILE  AVAILABILITY 

The  availability  of  a  system  subjected  to  a  sequence  of  calendar 
spaced  checkouts  is  considered.  Formulae  for  calculating  the  optimum 
frequency  of  checkout  are  given  for  the  situation  which  considers  checkout 
time  as  down  time.  Imperfect  repair,  imperfect  checkout,  and  resource 
limitations  are  treated.  A  technique  for  the  estimation  of  two  parameters 
of  the  availability  model  is  also  given. 

EXAMPLE  F  -  A  VULNERABILITY  MODEL  FOR  WEAPON  SITES 
WITH  INTERDEPENDENT  ELEMENTS 

This  example  describes  a  simple  "counting"  model,  employing 
probability  grid  transparent  overlays,  which  aid  in  the  determination  of 
the  trade-offs,  measured  in  survival  probability,  between  site  dispersal 
and  hardening  for  a  weapon  complex  composed  of  several  interdependent 
elements,  separated  by  distances  of  less  than  two  lethal  radii.  The  sur¬ 
vival-probability  expressions  are  obtained  through  the  use  of  Markov 
chains.  An  example  of  vulnerability  estimation  is  given. 
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APPENDIX  H 

EXAMPLE  OF  COST-EFFECTIVENESS 
ANALYSIS  FOR  THE 
DEFINITION  PHASE 
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EXAMPLE  OF  COST-EFFECTIVENESS  ANALYSIS 
FOR  THE  DEFINITION  PHASE 


1.0  INTRODUCTION 

It  is  the  purpose  of  this  example  to  illustrate  the  application  of  the 
principles  of  cost-effectiveness  analysis.  The  example  selected  illustrates 
the  type  of  simplified  analysis  that  occurs  early  in  the  Definition  Phase  of  a 
program  when  there  is  a  relative  dearth  of  data.  The  simplifications 
employed  are  compatible  with  the  accuracy  of  available  data. 

The  steps  in  this  example  correspond  roughly  to  the  first  nine  steps 
outlined  in  Section  II  of  this  report,  although  in  certain  instances  the  division 
is  not  explicit. 

It  is  not  the  purpose  of  this  example  to  illustrate  the  techniques  used 
in  estimating  reliability,  cost,  and  weight,  but  rather  to  show  how  such 
estimates  are  used  in  the  optimisation  process. 

2.0  LEVEL  OF  APPLICATION 

This  example  deals  with  the  third,  or  lower,  level  of  cost -effectiveness 
analysis:  that  of  optimisation  of  a  system.  The  two  higher  levels:  selection 
of  mission,  and  selection  of  system  are  assumed  to  have  been  completed. 

The  outputs  of  these  higher  level  analyses  have  defined  the  mission  and  the 
basic  system. 

3.0  PROGRAM  OBJECTIVES  AND  MISSION 

The  system  to  be  considered  in  this  example  is  an  orbiting  space 
station.  The  purpose  of  this  space  station  is  to  serve  as  a  manned  labora¬ 
tory  for  performing  experiments. 

The  program  objectives  require  the  completion  of  fifteen  separate 
experiment  tasks  or  functions.  Each  task  and  the  equipment  required  for 
the  conduct  thereof  is  assumed  to  have  been  defined.  The  tasks  are  inde¬ 
pendent.  Succrsr  criteria  have  been  established  defining  the  number  of 
successful  oper-ting  man-hours  required  to  complete  each  task.  The  suc¬ 
cessful  operation  of  the  experimental  equipment  for  the  full  duration  of  an 
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individual  mission  is  not  required.  If  the  equipment  fails  prior  to  completion 
of  the  planned  mission(only  the  uncompleted  portions  of  the  task  need  to  be 
run  on  a  subsequent  flight. 

The  planned  duration  of  each  mission  has  been  fixed  at  thirty  days. 

The  total  weight  of  the  experimental  equipment  to  be  carried  is  7670  pounds. 
The  total  time  required  for  completion  of  all  experiments  is  2160  man-hours. 

4.0  IDENTIFY  RESOURCES  AND  CONSTRAINTS 

4. 1  A  Priori  Resources  and  Constraints  It  is  assumed  that  prior 
optimisations  have  selected  the  booster  and  re-entry  vehicle  to  be  used  in 
the  program.  Selection  of  the  re-entry  vehicle  automatically  establishes 
the  number  of  crewmen  involved.  Resupply  or  ferry  operations  have  been 
established  as  being  uneconomical.  The  basic  station  configuration,  dimen¬ 
sions,  and  weights  are  given. 

Within  the  above  framework,  together  with  the  preceding  state¬ 
ment  of  pmgrsm  objectives,  resource-,  can  be  identified  and  constraints 
established. 

4.  2  Customer  Resources  and  Constraints  The  usual  resources 
which  the  customer  has  to  devote  to  satisfaction  of  his  needs  are  cost  and 
calendar  time. 

4.  2.  1  Cost:  Cost  is  a  customer  resource  which  is  always 
involved  in  a  program.  Bounds  on  cost  should  be  established  from  the 
higher  level  optimisations  as  those  cost  amounts  beyond  which  the  selected 
mission  .r  system  is  no  longer  competitive  with  alternative  missions  or 
systems. 

4.  2.  2  Calendar  Time:  In  this  example  the  experiment  results 
are  assumed  to  be  required  by  a  specified  date  in  order  to  support  develop¬ 
ment  schedules  for  other  programs.  Thus  calendar  time  is  stated  as  a 
constraint  or  bound. 

4.  2.  3  Crew  Skills:'  This  is  a  potential  resource  if  it  is 
established  that  use  of  in-flight  maintenance  is  a  feasible  or  desirable 
improvement  alternative.  The  maintenance  skill  of  t^e  customer-furnished 
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flight  crew  limits  the  selection  of  msintensnce  slternstives.  There  ere  two 
basic  bounds  here: 

(1)  the  level  of  skill  available  in  the  personnel 
inventory  without  additional  training  costs,  and 

(2)  the  level  of  skill  beyond  which  it  is  not  feasible 
to  go  because  of  calendar  limits  on  available 
training  time. 

4.  3  Mission  and  System -Imposed  Resources  and  Constraints 


Examination  of  the  program  requirements  indicates  that  mission  and  system 
imposed  upper  and  lower  bounds  exist  for: 

(1) 

weight  of  experimental  equipment  put  into 

orbit,  W 

0 

(2) 

power  required  to  operate  station 

(3) 

volume  required  to  accommodate  crew 
and  experiments 

(4) 

crew  time  to  accomplish  the  required 
functions,  T#. 

Since  power  an'  volume  can  be  expressed  in  terms  of  equivalent 
weight,  the  list  of  primary  resources  can  be  reduced  to: 

(1) 

(2) 

weight 

crew  time. 

Bounds  on  the  weight  resource  are  established  by  the  payload 
capability  of  the  preselected  booster. 

The  extent  of  the  crew  time  resource  is  determined  by  deducting 
requirements  for  such  items  as  eating,  sleeping,  and  recreation  from  the 
twenty-four  hours  available  each  day. 

5.0  SELECTION  OF  COST-EFFECTIVENESS  CRITERIA 

In  this  example  it  is  necessary  to  distinguish  between  single  mission 
effectiveness  and  total  program  effectiveness.  System  effectiveness  for  a 
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single  mission  may  be  defined  to  be  a  function  of  station  payload  capability 
and  the  probability  that  any  given  successfully  launched  booster  will  fly  the 
proper  trajectory;  inject  the  station  into  orbit;  and  that  the  station,  its  re¬ 
entry  vehicle,  and  its  experimental  payload  will  survive  for  a  specified 
period  of  time  in  the  proper  orbit.  Total  program  effectiveness,  on  the 
other  hand,  should  be  stated  in  terms  of  satisfying  the  program  objectives. 

For  the  system  covered  in  this  example,  the  program  requirements 
as  previously  stated  are  clear.  "The  objective  of  the  program  is  to  complete 
2160  man-hours  of  task  time,  which  requires  the  successful  orbiting  and 
functioning  of  7670  pounds  of  pc y load  equipment." 

No  probability  is  expressly  attached  to  these  objectives.  Accordingly, 
we  shall  regard  these  requirements  as  fixed  and  let  the  number  of  launches 
and  the  effectiveness  of  each  mission  vary  to  achieve  these  requirements. 

In  view  of  this  decision  we  shall  define  the  cost-effectiveness  criterion  to  be: 

Minimise  the  total  expected  program  cost  by  varying 
system  effectiveness,  subject  to  the  constraint  that 
sufficient  flights  must  be  accomplished  to  place  7670 
pounds  of  successfully  operating  experimental  pay- 
load  in  orbit  and  to  perform  2160  man-hours  of  experiment 
time. 

It  should  be  carefully  noted  that  this  implies  that  the  optimum  value  of 
system  effectiveness  is  that  value  which  minimises  the  expected  cost  of  the 
program.  There  is  a  calculable  risk  attached  to  attaining  the  program 
objectives  at  the  expected  cost,  a  fact:  which  we  shall  consider  in  a  later 
section. 

6  0  IDENTIFY  VARIABLES 

Variables  of  an  analysis  are  those  pirameters  or  quantities  the  use  of 
which,  when  varied,  will  result  in  variations  in  resources  or  the  effective¬ 
ness  with  which  the  prografa  objectives  are  accomplished.  In  the  present 
example  we  shall  make  use  of  the  following  parameters: 

Aj  =  Nonrecurring  cost  of  the  i*^  improvement  a  ternative 
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a!  =  Recurring  coat  of  the  i*k  improvement  alternative  on 
each  flight  attempt 

CQ  =  Nonrecurring  bate  coat  of  the  program 

c,  =  Recurring  coat  of  a  flight  attempt  excluding  the  recurring 
coata  of  atation  improvementa 

C{  =  Total  program  coat 

AC*  =  The  change  in  total  program  coat  reaulting  from  adding 
the  i^  alternative  to  the  primitive  ayatem 

E  =  Single -rniaaion  ayatem  effectiveneaa 

f .  =  Expected  number  of  flighta  required  for  the  j1**  experiment 

k  =  Fraction  of  atation  failurea  that  occur  in  orbit 

L  =  Mia  a  ion  duration  in  daya 

m 

-  Station  failure  rate  in  orbit 

M.  =  Average  man -hour  a  per  day  expended  on  in-flight 
maintenance  of  the  i™  improvement  alternative 

Nm  =  Expected  number  of  flight  attempts  required  to  complete 

a  program 

N(  =  Number  of  aucceaaful  flighta  required  to  complete  program 
=  Probability  that  launch  vehicle  will  be  aucceaaful 

pj1  =  Probability  that  re-entry  vehicle  will  aurvive  boo  at  {diaae 

pk  =  Probability  that  atation  will  aurvive  the  booat  phaae 

P^  =  Experiment  reliability 

P  =  Miaaion  reliability  of  rocket  launcher,  atation,  and  re-en*ry 
m  vehicle 

P *  =  Probability  that  re-entry  vehicle  will  aurvive  in  orbit 

Pa  -  Probability  that  atation  will  be  available  in  orbit  for  a 

*  Specified  length  of  time 

Q  =  (General  definition)  probability  of  failure 

lH 


r.  =  Reliability  of  a  subsystem  after  incorporation  of  the  i*h 

1  improvement  alternative 

r.°  =  Reliability  of  the  subsystem  prior  to  incorporation  of 

an  improvement  alternative 

Rj  =  Reliability  cf  the  j**1  experiment 

R(  =  Total  station  reliability  for  a  time  duration 

R(  =  Reliability  of  "stripped"  station 

o 


Reliability  of  station  in  orbit  for  a  time  period  Lm 


=  Time  available  for  experiment  work  per  successful  flight 
(man-hours) 

Tc  =  Time  available  for  experiment  or  maintenance  work  per  day 
(man-hours)  based  on  "stripped"  station 

Te  =  Equivalent  total  man-hours  required  to  complete  all 
experiments 

Tjj  =  Total  man-hours  required  to  complete  all  experiments 

tj  =  Planned  start  time  of  the  ^  experiment  measured  from 

start  of  station  orbit 

t2  =  Planned  stop  time  of  j**1  experiment  measured  from  start 
of  station  orbit 

7.  =  Expected  time  to  station  failure  in  time  L_ 

x  m 

=  Totai  equivalent  weight  of  all  experiment  or  payload  hard¬ 
ware  in  program 

Wh  =  Weight  of  the  i**1  improvement  alternative 

» 

Wj  Weight  of  the  experiment 

Wp  =  Experiment  payload  per  flight 

W  =  Payload  capability  of  stripped  station 

Po 

Z.  Ranking  index  (dollars  change  in  program  costs  divided  by 

weight  increase  of  station). 

In  view  of  the  definition  of  a  variable  given  above,  virtually  all  of 
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these  parameters  may  be  regarded  as  variables.  In  the  present  example  we 
shall  treat  the  parameters  W^j,  r.,  and  Mj  as  variables.  The  remainder 
of  the  parameters  will  be  treated  as  constraints  except  insofar  as  they  are 
altered  by  one  of  the  three  above. 

7.0  DEVELOP  RELATIONSHIPS  AND  MODELS 

7.  1  Method  of  Approach  The  analysis  commences  with  a"stripped" 
orbiting  station.  This  primitive  system  consists  of  the  least  amount  of 
hardware  required  to  accomplish  the  objectives  of  the  program.  It  is  the 
least  costly  system  to  build;  it  weighs  the  least;  and  it  provides  the  maxi¬ 
mum  amount  of  available  experiment  payload  weight  since  the  difference 
between  the  rocket  booster  payload  capability  and  the  station  and  crew 
weights  represents  the  maximum  available  experimental  payload  that  may 
be  loaded  aboard  the  station.  Unfortunately,  this  primitive  system  is  also 
the  least  reliable.  Thus,  from  the  point  of  view  of  the  cost  of  accomplishing 
the  fixed  program  objectives,  it  is  apt  to  prove  to  be  less  satisfactory  than 
a  more  costly  laboratory  having  less  payload  capability.  The  reason  for  this 
is  that  a  low  station  reliability  implies  a  larger  number  of  required  flights 
to  attain  the  fixed  program  objectives.  On  the  other  hand,  station  reliability 
improvement  can  only  be  attained  by  adding  station  equipment  redundancy 
which  implies  an  increase  in  station  weight  and  a  resultant  decrease  in  the 
weight  that  is  available  for  the  experiment  payload.  Thus  an  increase  in 
reliability  can  result  in  either  a  decrease  in  required  number  of  flights  or 
an  increase  in  the  required  number  of  flights  depending  upon  whether  the 
increase  in  reliability  decreases  the  expected  number  of  flight  failures 
faster  than  the  attendant  increase  in  station  weight  increases  the  need  for  a 
larger  number  of  flights  because  of  reduced  experiment  payload  capability. 
The  equations  which  we  shall  shortly  proceed  to  develop  will  relate  the 
economics  of  improving  the  reliability  of  the  station  to  the  cost  of  performing 
additional  flights.  Station  reliability  will  be  a  parameter  which  will  be  per¬ 
mitted  to  vary  as  we  seek  a  minimum  cost  program  that  will  meet  the  fixed 
program  objectives. 

As  was  indicated  earlier,  we  shall  seek  to  minimize  the  expected 
cost  of  the  program.  We  shall  calculate  this  expected  cost  on  the  basis  of 
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the  expected  number  of  flight  attempts  required  to  achieve  2160  mar. -hours  j 
of  experiment  time  or  7670  pounds  of  experiments  in  orbit,  whichever  repre¬ 
sents  the  worst  constraint,  thus  the  total  cost  of  the  program  will  be 
expressed  as: 

C.  =  (expected  number  of  flight  attempts)  x  (recurring 
cost  of  a  flight)  4  (nonrecurring  program  cost) 

This  function  will  approach  a  minimum  value  and  then  increase 
as  the  reliability  of  the  station  is  increased.  The  minimum  value  corre¬ 
sponds  to  the  "optimum"  choice  of  station  configuration.  It  should  be  care¬ 
fully  noted  that  this  minimuru  is  not  an  abfiolute  minimum;  it  is  only  best  in 
the  sense  that  it  reflects  the  best  choice  among  a  given  finite  set  of  alterna¬ 
tives  proposed  by  a  given  design  group.  Lower  cost  may  in  fact  be  achievable 
by  other  designs. 

7.2  Mission  Success  Probability  As  noted  earlier,  system  effec¬ 
tiveness  may  be  defined  as: 

E  =  f[P.  P  .  Wj  (1) 

m  e  p 

where 

P  =  mission  success  probability  of  rocket  launcher, 

station,  and  re-entry  vehicle 

Pg  =  experiment  reliability 

Wp  =  station  payload  capability. 


W e  shall  not  write  a  specific  expression  for  E,  but  instead  will  treat  the 
components  P^,  P^,  and  separately.  Accordingly,  we  define: 


K 

P,  =  "  Rj-  (2) 

e  i  =  1  J 


R.  =  reliability  of  the  j**1  experiment  of  a  flight  of  K 
experiments 

and, 

P  =  P.  Pb  P*  Pb  P‘  (1) 

m  b  s  s  r  r 
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where 

=  probability  that  launch  vehicle  will  be  successful 
P*  =  probability  that  station  will  survive  the  boost  phase 

P*  =  probability  that  station  will  be  available  in  orbit  for 

*  a  specified  length  of  time 

pk  =  probability  that  re-entry  vehicle  will  survive  boost 
r  phase 

P*  =  probabil„y  that  re-entry  vehicle  will  survive  in  orbit 


Although  may  be  regarded  as  a  constant  from  flight  to 

flight,  P  cannot  be.  Therefore  we  shall  account  for  P  in  terms  of  an 
©  © 

equivalent  number  of  required  successful  station  flights. 


7.  5  Expected  Number  of  Flights  Let  it  be  assumed  that  Ng  suc¬ 
cessful  flights  are  required  and  that  the  probability  of  achioving  a  successful 
flight  is  a  constant  p.  The  expected  number  of  flight  attempts  N&  required 
to  achieve  N#  successes  is  given  by: 


Na  “  Ng  {l  •  p +2  (1  -  p)p  +  3  <1  -  p)2  p  + 


r 


N 

s 


m 

E  kp(l-p)k 
k  =  1 


} 


(4) 


We  must  now  relate  p  and  to  experiment  weight,  launch 
booster  payload  capability,  and  system  effectiveness. 


Consider  N  .  The  program  is  not  complete  until  all  experi¬ 
ments  have  been  conducted.  This  requires  a  number  of  totally  successful 
flights  N#  determined  by  either  man-hours  available  for  conduct  of  experi¬ 
ments  per  flight  (T^)  or  by  experiment  payload  (W^)  per  flight.  Thus  the 
number  of  flights  is  given  by  either: 


N 


T 

* 


(5a) 
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or  by 


(5b) 


whichever  is  greater,  where 

Te  =  total  equivalent  man-hours  of  experiment  time  required 

T  =  total  man-hours  of  experiment  time  available  per 
*  flight 

W  =  total  equivalent  weight  of  experiments  in  the  total 
e  program 

W  s  total  experiment  payload  weig  '  that  can  be  carried 
P  per  flight. 

The  time  available  per  flight  (T#)  in  man-hours  is  given  by: 


E  Mi>  Lm 
i  r  1  1  ra 


where 


time  available  per  day  for  either  experimentation  or 
for  additional  station  maintenance  activities  based  on 
stripped  station 

average  man-hours  per  day  expended  on  the  i**1 
improvement  alternative  to  the  stripped  station 

mission  duration  in  days. 


We  may  determine  T  from  the  following  considerations.  Let 
the  reliability  of  the  j**1  experiment  be  R^.  We  shall  assume  that  each 
experiment  is  scheduled  for  two  flight  attempts  at  most.  The  second  flight 
is  required  only  if  it  fails  its  first  flight.  Further,  the  man-hours  expended 
on  an  experiment  that  fails  are  not  wasted.  Only  that  portion  of  the  experi¬ 
ment  not  completed  needs  to  be  conducted  on  the  subsequent  flight.  In  view 
of  these  assumotions  the  expected  number  of  flights  f.  required  for  the  j*b 
experiment  is: 

f .  =  1  •  Rj  +  2(1-  R  )  P) 

=  2  -  \ 
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Hence  if  the  time  required  to  complete  all  experiments  is  T,^  we  may  write 
equation  (5a)  in  terms  of  equations  (6)  and  (7)  as: 

NT  =  - la -  (£  E  (2-R,)).  (8) 

(Tc  -  i  *  *i)  Lm 
For  equation  (5b)  we  have  that 

K 

W  =  W  -  E  W.  (9) 

P  po  i  =  1  hi 

where 

W  =  experiment  payload  capability  of  the  stripped  station 

Po 

TIL  a  weight  of  the  i**1  improvement  alternative  to  the 
Hi  stripped  station. 

If  the  jth  experiment  weighs  pounds,  we  must  be  prepared 
to  carry  a  total  equivalent  payload  W  of 


W  «  E  (2  -  R.)  W 

*  j  *  1  J  J 

Thus  equation  (5b)  becomes  N 

E  (2  -  R.)  W 

NW  .  J-L> - Li  . 

*  K 

W  -  E  W. 
i*l  hi 

Strictly  speaking,  the  R^  should  be  averages  obtained  by 


(j  .  J2  X.c-  V  -•l1"* 

t  t 


where 


station  failure  rate  in  orbit  (assumes  exponential 
failure  distribution  for  station) 
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Rj{t)  =  reliability  of  the  j**1  experiment 

t,  =  planned  start  time  of  j**1  experiment  measured  from 
start  of  station  orbit 

1 2  =  planned  stop  time  of  j**1  experiment  measured  from 

start  of  station  orbit. 

However,  for  our  present  purposes  it  is  sufficiently  accurate  to 
ignore  this  refinement  of  the  analysis  and  simply  regard  the  as  con¬ 
stants  independent  of  the  station  failure  :rate  in  orbit  (X^). 

Since  we  have  accounted  for  experiment  reliability  in  W  and  T  we 

may  identify  p  of  equation  (4)  with  P  of  equation  (1).  Thus  we  have 

m 


N  = 
a 


whichever  is  the  greater. 

Our  development  is  not  complete  until  we  have  examined  P  . 

s  ® 

We  require  an  expression  for  the  factor  P^  o*  Pm  which  accounts  for  the 
fact  that  a  station  which  fails  in  orbit  will,  in  general,  have  performed  some 
useful  service,  ft  is  assumed  that  its  usefulness  will  be  linearly  proportional 
to  the  time  nonfailed,  thus  P*  is  given  by 


where 


Tf  =  expected  time  to  station  failure  in  time 


scheduled  mission  duration. 


If  it  is  assumed  that  the  station  fails  exponentially  in  orbit  at  a 


rate  X#,  then, 


■  J. 


-  X  t  . 
e  s  dt 


.  -XL 
1  -  e  s  m 
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where  e~  is  the  total  station  reliability  for  a  time  *t".  Thus, 


„s  1  -  e"  XsHn 

p.  =  — rr - 


s  m 


If  the  station  reliability  is  reasonably  high, 


.  -X  L 
1  -  e  s  m 

*7w“ 


L  +  (XL) 
m  s  m 


TT, 

s  m 


(16) 


(17a) 


=  1  - 


X.Lm 


(17b) 


which  to  the  same  order  of  approximation  is 


1  - 


X  L 

s  m 


-  X.Lm>  + 


1  -  Cl  -  x. L  ) 


s  m 


(17c) 


~  R 

X 


R  *  =  reliability  of  station  for  a  time  period  L_  in  orbit 

i  m 

Q*  *  1  -  R  *  for  same  time  period. 


Thus  we  may  write  PR  to  a  first  order  approximation  as 


Q* 


Pm  =  PbP.  Pr  Pr’  <R.‘  *  “f  > ' 


(18) 


If  we  define  mission  unreliability  Q  as 

m 


Q  4  1  -  P  Pb  Pb  P*  R  * 

m  b  s  r  r  s 


(19) 


121 


« 


then  P  may 
m 

be  expressed  as 

p 

m 

1  -  <°n,  •  q, p?  P‘  11  Q.> 

(20) 

where 

ko,  4  Q; 

(21) 

k 

s 

fraction  of  station  failures  that  occur  in  orbit 

Q. 

s 

total  station  unreliability  =  1  -  R# 

R 

s 

s 

total  station  reliability  for  a  time  duration  L  . 

m 

We  shall  assume  that  the  station  reliability  R#  may  be  ex¬ 
pressed  in  terms  of  the  stripped  station  reliability  Rs  by  the  series 

relationship  ° 

K 

R  *  R  n  ri  (22) 

*  o  i  =  1  “o 


*i 


where 


r.  =.  the  reliability  of  the  ith  improvement  alternative 

r,°  *  the  reliability  of  the  i***  "subsystem"  before 

incorporation  of  the  i**»  improvement  alternative. 

Note  that  this  equation  .equires  that  when  two  successive  station  improve¬ 
ments  are  made  to  one  subsystem,  the  second  improvement  must  either 
cancel  the  first  or  else  be  defined  in  terms  of  a  new  r.°. 

7.  4  Development  of  Cost  Equation  We  are  now  in  a  position  to 
write  down  the  total  coat  C{  of  the  program. 


whor<* 


K 

E 

i  =  1 


N.  i.cf  *  , 1 .  Az) 


K 

r 

i  =  1 


Ai  + 


(23) 


the  recurring  coat  of  a  flight  attempt  excluding  the 
recurring  coata  of  station  Improvement 
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A,1 

L 


-  the  recurring  cost  of  the  station  improvement  on 
each  flight  attempt 

=  the  nonrecurring  cost  of  the  i**1  station  improvement 
=  the  nonrecurring  base  cost  of  the  program 


whichever  is  greater,  and 


TN  fi  = 


(t  -  s  m  .)l 

\  c  ,  ,  t/  r 


(24a) 


£  (2  -  R.)  W 

,W  j  =  1  }  3 


W  -  Z  W. 
Po  i  =  1  hi 


1  -  (Qm  -  Pv  P*  p'  p.b  b  QJ 

m  o  r  r  s  s 


i  *  Q. 


(24b) 


Equations  (23)  through  (26)  are  all  that  are  required  to  per¬ 
form  the  required  analysis. 

8.  0  IDENTIFY  ALTERNATIVES  AND  DEVELOP  MODEL  INPUTS 


As  noted  earlier,  the  first  step  used  here  is  to  establish  a  basic 
stripped  down,  or  primitive,  design  which  meets  functional  requirements 
with  absolute  minimum  expenditure  of  resources.  This  design  is  minimum 
weight,  minimum  cost,  and  is  stripped  of  all  redundance,  in-flight  spares, 
or  other  improvements. 


Since  the  system  meets  functional  requirements,  improvement  alter¬ 
natives  are  generally  confined  to  items  which  improve  station  reliability. 
Types  of  alternatives  considered  are  shown  in  Table  I. 

For  each  station  subsystem,  various  levels  and  types  of  redundancy 
and  in-flight  maintenance  (Table  I)  are  considered.  Between  six  and  twelve 
such  alternatives  should  be  established  for  each  subsystem.  Usually  this 
will  be  accomplished  by  the  individual  design  groups  responsible  for  the 
subsystems  involved. 

The  definition  of  each  alternative  should  include: 

(1)  schematic  showing  functional  relationships 

(2)  description  of  any  required  cross-connecting  or 
monitor  /transfer  devices,  and 

(3)  failure  mode  and  effect  analysis. 

For  maintenance  items,  the  definition  includes  Additionally: 

(1)  description  of  required  maintenance  actions 

(2)  listing  of  required  tools  and  fault  isolation  equipment 

(3)  notation  of  any  repackaging  required  to  permit 
maintenance,  and 

(4)  description  of  any  crew-member  training  needed  to 
perform  maintenance. 

Table  II  is  an  example  of  alternatives  considered  for  the  electric 
power  subsystem  of  the  station  which  is  diagrammed  in  Figure  1 .  The 
basic,  or  primitive,  subsystem  contains  two  fuel  cell  modules,  both  of 
which  must  continue  to  function  throughout  the  mission  in  order  to  achieve 
mission  success.  The  reliability  of  this  subsystem  is  64  per  cent  (r®  of 
Table  II). 

The  i.d.  column  of  this  table  lists,  by  code  number,  the  alternatives 
which  are  being  considered  for  this  subsystt.n.  These  alternatives  are 
described  as  follows: 


TABLE  I 


IMPROVEMENT  ALTERNATIVES 

A.  BASIC  FAILURE  RATE  REDUCTION 

B.  REDUNDANCY 

1.  LEVEL 

(a)  Component 

(b)  Subsystem 

2.  TYPE 

(a)  Dual  Rom  With  Manual  Monitor  and  Transfer 

(b)  Dual  Rom  With  Automatic  Monitor  and  Transfer 

(c)  Duality  of  Several  Rems  With  and  Without 
Intermediate  Crossovers. 

3.  OPERATING  MODE 

(a)  Standby 

(b)  Continuous  Operating 

C.  IN-FLIGHT  MAINTENANCE 

1.  TYPE 

(a)  Adjustment 

(b)  Remove  and  Replace 

(c)  Repair 

2.  CONFIGURATION;  MAINTENANCE  TO: 

(a)  Single  Thread  Systems 

(b)  Redundant  Systems.  As  Listed  Above 

(1)  Using  On-Board  Spares 

(2)  Cannibalising  Parallel  Systems 


D.  COMBINATIONS  OF  THE  ABOVE 


TABLE  II 
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a-1  Provide  one  redundant  fuel  cell  module  making  a  total  of  three,  ( 
with  only  two  required  to  satisfy  the  power  load. 

a-2  Add  second  redundant  fuel  cell  module  making  a  total  of  four, 
with  only  two  required  to  satisfy  the  power  load. 

a-3  Add  redundant  D.C.  regulator  to  power  distribution  subsystem. 

a-4  Add  redundant  inverter  in  power  distribution  subsystems. 

a-5  Add  a  redundant  fuel  cell  reactant  control  package. 

a -6  Provide  one  redundant  set  of  functional  components  in 
electrical  load  panels. 

a-7  Add  one  redundant  set  of  functional  components  in  junction  boxes. 

a -8  Provide  tools  and  spare  parts  for  in-orbit  maintenance  of  power 
distribution  subsystem. 

a-9  Provide  tools  and  spare  parts  for  maintenance  of  reactant 
controls. 

The  r.°  column  indicates  the  reliability  level  of  the  stripped-down 
subsystem.  The  r.  column  indicates  the  subsystem  level  which  would 
result  from  incorporation  of  the  alternative,  except  in  the  case  of  alternative 
a-2.  In  this  latter  case  the  r.°  column  contains  the  combined  reliability  of 
the  fuel  cells  and  the  r.  column  shows  the  increase  in  subsystem  reliability 
due  to  fuel  cells  alone. 

Columns  Aj  and  A.,  indicate  the  cost  of  the  hardware  ipvolved  in  the 
improvement.  For  redundant  items,  the  cost  includes  the  design  and  devel¬ 
opment  cost  of  the  redundant  item  and  its  associated  monitor /transfer  and/or 
cross  connection  devices,  plus  the  recurring  cost  of  the  additional  procured 
items.  The  cost  of  maintenance  items  includes  the  cost  of  any  repackaging 
of  equipment  necessary  to  permit  maintenance,  together  with  the  cost  of  the 
spare  parts,  tools,  fault-isolation  equipment  and  any  training  required  to  en¬ 
able  the  crewmen  to  successfully  perform  the  specified  maintenance.  Aj  is 
the  nonrecurring  portion  and  A^  is  the  recurring  portion  of  this  cost. 

The  column  irdicat*  s  the  amou  it  of  weight  expended  in  order  to 
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incorporate  the  improvement.  In  the  case  of  redundant  elements,  the  weight 
figures  include  the  additional  hardware,  plus  any  monitor /transfer  or  other 
interconnecting  elements  required  to  make  the  redundant  clement  function. 
For  maintenance  items,  the  weight  figures  include  the  spare  parts,  tools, 
and  fault -isolation  equipment  needed  to  accomplish  the  maintenance. 

Column  M  indicates  the  estimated  average  maintenance  man-hours 
per  day  expended  on  proposed  maintenance  items.  This  value  is  calculated 
as  the  estimated  maintenance  man-hours  for  each  occurrence  of  failure 
multiplied  by  the  probability  that  said  failure  will  occur. 

The  Z  column  shows  the  ranking  index  for  each  alternative  which  is 
the  cost  reduction  calculated  from  equation  (23),  divided  by  the  weight 
expended  to  incorporate  the  alternative.  The  reason  for  this  index  will  be 
discussed  in  the  next  section. 


As  noted  in  the  introduction,  it  is  not  the  purpose  of  this  example  to 
illustrate  the  development  of  model  inputs,  but  rather  to  illustrate  the  use  of 
inputs  in  arriving  at  an  optimum  cost-effectiveness  solution.  The  reader  is 
referred  to  the  examples  of  Task  Group  II  and  the  open  literature  for  guid¬ 
ance  in  the  estimation  of  parameters.  We  shall  simply  assume  here  that 
the  reliability,  weight  and  cost  of  each  alternative  and  of  the  primitive  system 
and  the  experiments  have  been  assessed.  The  net  result  for  each  case  is 
shown  in  Tables  ID,  IV  and  V.  The  information  in  these  tables  constitutes 
the  data  base  for  exercising  equations  (23)  through  (26). 

9.°  EVALUATION  OF  ALTERNATIVES 

9.  1  Determination  of  Nfl  As  noted  earlier  is  given  by  either 
(24a)  or  (24b)  whichever  is  greater.  In  the  first  instance  we  have,  using 
the  data  of  Tables  II,  IV,  V  and  VI 


N 


2289.6 

(18  -E'MTTO' 


(27) 


W 


The  alternative  expression,  is  from  Tables  III  and  VI  and 

equation  (4): 

1*  O  lift 

(28) 


V. 


8360 


TABLE  HI 


CHARACTERISTICS  OF  THE  PRIMITIVE  SYSTEM 

CQ  =  $250  x  10^  =  program  nonrecurring  base  coat 

C|  =  $60  x  10^  per  flight  attempt 

W  a  3500  pounds  (launch  vehicle  limited) 

Po 

T  *  18  man-hour ■ /day 

L  =30  days 

m 

Pb  =  0.85 

Pj*  *  .992 

R*  =  0.0372 

“o 

P‘  =  0.9936 

PrPr  55  0975 

T  =2160  man-hours 

e 

k  =0.92  (assumed  constant  for  all  system  improvements) 
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TABLE  IV 


CHARACTERISTICS  OF  THE  EXPERIMENTS 


Experiment 

Number 

Weight 

(pounds) 

Reliability 

1 

1400 

.85 

2 

1200 

.87 

3 

200 

.96 

4 

500 

.94 

5 

650 

.94 

6 

350 

.97 

7 

350 

.93 

8 

300 

.95 

9 

900 

.90 

10 

550 

.90 

11 

250 

.96 

12 

600 

.95 

13 

150 

.98 

14 

70 

.97 

IS 

200 

.99 

m 


TABLE  V 


CHARACTERISTICS  OF  THE  ALTERNATIVES 


Dr*  ignitor 

Reliability 
of  Stripped 
Subeyatem 

< 

Reliability 
of  Improved 
Subsystem 

ri 

Weight  of 
Improvement 

<w.  ) 

*1 

in  pound* 

Maintenance 
Time  Required 

M{ 

in  man-hour* 

Recurring 
Coet  . 

A2 

in  10*  dollar* 

Nonrecurring 
Co*t  . 

A1 

in  10*  dollar* 

bb-) 

.4914 

.941) 

4.4 

.004) 

.00)4 

.00) 

bc-1 

.70979 

.94) 

14 

0 

.0144 

.004 

4-1 

.  9179 

.94v2 

90 

0 

.2941 

.14 

«  -  J 

.4242 

.9917 

194 

.0)2 

.179 

.09 

be -2 

.94) 

.9944 

12.7 

.0102 

.0122 

.004 

bd-4 

.9291 

.9)11 

2 

0 

.00) 

.001 

.441 

.442 

4.1 

.0014 

.0211 

.014 

f-4 

.794* 

.4109) 

24 

.00194 

.0241 

.011 

1*1 

.441 

.9442 

144 

0 

.  1971 

.04 

(-2 

.•109) 

.9444 

100 

0 

.2447 

.01 

be-1 

.44144 

.99404 

100 

0 

.1229 

.04 

1-40 

.441 

.44404 

4.9 

0014 

.0402 

.04 

a-6 

.441 

.4497 

20 

0 

.042 

.044 

bb-6 

.4414 

.44171 

)2 

0 

.0144 

.004 

b»  -  J 

.9197 

.99)4 

170 

0 

.1149 

.044 

*-• 

.441 

.4442 

14 

0 

.0411 

.0) 

bd-l 

.9251 

.924 

12 

0 

.0044 

.00) 

ba-60 

.9197 

.  924)4 

14.7 

.00044 

.0)42 

.014 

a-7 

.441 

.4422 

14 

0 

.024 

.024 

bb-7 

.4414 

.44244 

24.1 

.0097 

.0019 

.001 

c  - 1 

.99 

.994 

140 

0 

.0004 

.000) 

WO 

.97704 

.9974 

424 

0 

.  1497 

.06 

bl  -40 

.99)4 

.9994 

140 

0 

.0979 

.04 

e-2 

.994 

.999 

400 

0 

.0014 

.0009 

>•2 

.99402 

.99992 

144 

0 

.  147) 

.0) 

b«-20 

. •4144 

.444)1 

1400 

0 

.494 

.4 

t-4 

.441 

.4421 

20 

0 

.0444 

.02 
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EQUIVALENT 

WEIGHT 

1610  ! 

1360  | 

208  i 

530  | 

690  ! 

360  1 

375  is 

315 

990 

S09 

260 

630 

153 

M 

r» 

zoz 

EXPECTED 
FLIGHT 
(2  -  R  ) 

»T> 

• 

*■4 

1.  13 

1.00 

vO 

o 

• 

vO 

o 

• 

P-4 

1.03 

t» 

o 

• 

i-4 

1.05 

1.  10 

o 

<—4 

• 

(—4 

1.04 

1.05 

1.02 

1.03 

1.01 

RELIABILITY 

Rj 

S8* 

.87 

O' 

• 

♦ 

O' 

• 

T 

O' 

• 

r~ 

O' 

• 

fO 

a- 

• 

! 

o 

O' 

• 

o 

O' 

• 

vO 

O' 

4T> 

a* 

• 

CO 

O' 

• 

r- 

O' 

• 

.99 

INSTALLED 

WEIGHT 

wj 

1400 

1200 

200 

500 

650 

o 

IT 

350 

300 

006 

550 

250 

009 

o 

in 

•—4 

70 

200 

H 

Z 

U  ee 

§« 

w? 

& 

u 

IT 

sO 

- 

r~ 

00 

O' 

o 

•m 

*■* 

•m 

♦ 

IT 

•m 

ns 


7670  pounds  W  =  TOTAL  EQUIVALENT  =  8360  pounds 

e  WEIGHT 


9. 2  Ranking  the  Improvements  Table  V  lists  27  alternatives  that 
must  be  considered  in  evaluating  equation  (23).  If  we  approach  this  task  in 
a  brute  force  manner  we  would  first  evaluate  for  the  primitive  system 
and  then  again  for  each  of  the  27  alternatives,  one  at  a  time.  The  next  step 
would  be  to  incorporate  into  the  station  that  altsrnative  which  yielded  the 
greatest  reduction  in  C(  for  the  least  expenditure  of  We  would  now 

have  26  alternatives  to  evaluate  again  using  the  new  improved  station  reli¬ 
ability.  It  is  clear  that  this  process  requires  that  379  separate  calculations 
be  made  —  not  an  inconsiderable  amount  of  labor.  I  our  computation  pro¬ 
cess  is  to  be  efficient  it  is  desirable  to  find  a  short  cut.  Therefore  we  shall, 
in  fact,  procesd  as  follows. 


First,  we  shall  calculate  the  change  in  C(  (denoted  by  ACt)  for 
each  of  the  27  alternatives  applied  one  at  a  time  to  the  primitive  system. 
Second,  we  shall  rank  sack  AC.  in  accordance  with  its  efficient  use  of  the 

t  — i — <  «.  

weight  reserve.  That  is,we  shall  use  a  ranking  index  Z{  defined  as—  , 


Z 


l 


a  dollars 
pounds 


(29) 


1C.1  «  the  change  in  total  program  cost  resulting  from  adding 

the  alternative  to  the  primitive  system 

W  *  the  weight  of  the  i**1  alternative. 

••2 


The  logic  of  this  approach  is  contingent  upon  the  fact  noted 
earlier;  that  weight  is  a  resource  which  has  value.  Thus,  in  installing 
improvement  alternatives,  it  is  necessary  to  install  these  alternatives  in  an 
order  which  will  permit  the  maximum  possible  amount  of  cost  reduction  per 
unit  of  expenditure  of  this  resource. 

We  shall  then  incorporate  sufficient  alternatives  in  the  order  of 
thie  ranking  until  the  station  reliability  has  substantially  increased.  Then 


U 


Note  that  Zj  is  positive  if  ■  cost  decrease  occurs  when  the  alternative  is 
incorporated,  and  nee  stive  if  the  cost  increases. 


we  shall  rerank  the  remaining  alternatives  using  the  augmented  reliability  of 
the  station  as  the  new  basis  for  comparison  of  the  effect  of  incorporating 
alternatives  one  at  a  time.  There  is  a  danger  in  this  short  cut  approach 
which  is  illustrated  in  Figure  2.  This  figure  illustrates  the  manner  in  which 
Z.  changes  as  R  *  changes.  Note  that  alternative  a -8  is  worse  than  use¬ 
less  when  R  *  -  .  125,  but  for  larger  values  it  may  prove  to  be  valuable. 

The  saving  element  here  is  that  all  the  tend  to  increase  as  R  *  increases 
so  that,  in  practice,  several  alternatives  can  be  incorporated  before  the 
basic  station  reliability  is  increased  by  a  sufficient  amount  to  radically  alter 
the  ranking.  For  the  present  example  there  are  sufficient  alternatives  to 
bring  the  station  reliability  to  levels  in  excess  of  90  per  cent.  Accordingly, 
the  alternatives  which  remain  after  the  station  reliability  has  reached  50  per 
cent  will  be  re-evaluated  based  upon  a  revised  "initial"  station  reliability  of 
70  per  cent,  a  value  which  was  midway  between  the  50  per  cent  and  90  per 
cent  levels.  The  remaining  alternatives  which  proved  favorable  will  be 
retained  based  on  the  values  of  the  second  ranking. 

It  must  be  remembered  that  once  an  alternative  is  incorporated, 
some  of  the  other  alternatives  may  be  rendered  technically  infeasible  or  the 
amount  of  reliability  improvement  gained  by  them  may  be  radically  altered 
due  to  the  functional  relationships  between  the  various  alternatives.  There¬ 
fore,  during  the  ranking  process,  the  functional  interfaces  and  interferences 
between  alternatives  must  be  kept  in  mind.  Alternatives  must  be  redefined 
and  re-evaluated  as  required  by  prior  incorporation  of  interfacing  compo¬ 
nents.  For  example,  alternatives  a-5  and  a-9  as  described  in  paragraph 
8.0  above,  are  so  related  that  when  a-5  is  incorporated  a-9  is  no  longer 
profitable.  Thus,  a-5  is  redefined  to  include  cancellation  of  a-9  and  is 
identified  a»  a -50  in  Table  V. 

9.  3  Model  Exercise  We  shall  illustrate  the  use  of  the  equations 
developed  herein  in  detail  for  one  alternative.  Consider  the  first  alternative 
(bb-3)  of  Table  V.  From  equation  (29), 

.  Cost  of  program  using  stripped  station  -  Cost  of  pro- 
,  _  AC.1  gram  using  station  with  first  alternative  incorporated  (30a> 
L\  ^  =  -  weTgK  oTTfiFiF  alternative -  - 
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From  equation  (2?)  and  Tables  III  and  VI, 


(30b) 


M  equal  the 
•o  greater  of 


Te 

2289.6 

=  4.24 

(31a) 

t  L  " 

18  x  30 

c  m 

W 

e  = 

8360 

=  2.39 

(31b) 

ur~ 

Tsmr 

Hence, 


and  from  equation  (28), 


N  equal  the 
al  greater  of 


Hence, 


N 


4.24 


2289.6 


(31c) 


(T  -  M.J  L - (18  -  .0053)50 

a  i  m 


=  4.24  (32*) 


W 


Po  hi 


^006-°8.6  =  239S-  <32b> 


N 


"I 


4.24 


(32c) 


For  purposes  of  preliminary  ranking  we  may  s*rt 


N 


N 


(32d) 
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and  to  further  simplify  the  calculations,  it  is  convenient  to  make  the 
additional  coarse  approximation 

Pb  Pj*  P*  P#b  =1  (33) 

Now,  from  equation  (26)  and  equation  (1), 


(34a) 


.0307 

'  75TTT 


.9813 


.0463. 


(34b) 


Hence, 
Z.  = 


4  x  60 
8:  6  (.0307 


x  106 

r.92  x 


7WBT 


4  x  60.0054  x  106 

8.6  {.0463  4  .92  x 

- Z -  (35) 


.003  x  106 

- 875 — 


8367  10^  dollars  change  in  program  costs 

x  “jght  increase  in  station 


This  calculation  is  repeated  until  all  the  alternatives  have  been 
evaluated.  Then  the  alternatives  are  arranged  in  descending  order  of  Z.. 

Next  we  assume  that  we  have  incorporated  cufficient  of  these 
ranked  alternatives  to  raise  the  station  reliability  to  about  .  50.  It  is  now 
necessary  to  rerank  the  remaining  alternatives,  since  as  wr»  noted  in 
Figure  1,  we  may  now  have  serious  errors  in  our  ranking.  Accordingly, 
the  alternatives  which  remain  after  the  station  reliability  has  reached  .  5 
are  re-evaluated  based  upon  a  revised  "initial"  station  reliability  of  .7, 
a  value  approximately  midway  between  .  5  and  the  total  potential  reliability 
of  .95  that  is  achievable  using  all  the  alternatives.  The  results  of  this 
second,  and  final,  ranking  are  shown  in  Table  VII. 

The  total  ranking  list  tor  the  station  produced  by  the  addition  of  the 
first  and  second  ranking  iterations  will  show  discontinuities  in  Z.  at  the 
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TABLE  VII 


ADJUSTED  RANKING  OF  THE  ALTERNATIVES 


Alternative 

R 

(i) 

m 

0 

.0307 

bb-3 

.0463 

bc-1 

.06155 

d-1 

.1172 

e-3 

.2725 

be -2 

.2882 

bd-4 

.2907 

a -9 

.2957 

f-4 

.317 

a-1 

.467 

f-2 

.558 

be-1 

.652 

a -50 

.659 

a -6 

.6682 

bb-6 

.679 

ba-3 

.733 

a-8 

.739 

bd-1 

.741 

ba-60 

.7445 

a -7 

.746 

bb-7 

.748 

c-1 

.7515 

bf-3 

.767 

ba-40 

.772 

c-2 

.775 

a  >2 

.780 

be-20 

.782 

a -4 

.783 

R  Z. 

a  t 


0372 

056 

.8367 

07445 

.4790 

142316 

.2363 

331146 

.2735 

350038 

.3554 

353065 

.3715 

359124 

.2609 

384811 

.2543 

567431 

.2339 

677897 

.  1906 

791846 

.  1808 

800568 

.1847 

811434 

.0701 

824276 

.0583 

890419 

.0532 

897643 

.0458 

900457 

.0298 

904040 

.0194 

905733 

.0054 

907791 

.0043 

912375 

,0042 

931761 

.0035 

937295 

.0016 

941063 

.0010 

946648 

-.0003 

949404 

-.--25 

951034 

-.0069 

Ml 


point  where  the  second  ranking  was  started.  This  discontinuity  is  reduced 
by  reranking  of  alternatives  in  the  area  of  discontinuity.  All  discontinuities 
cannot  be  eliminated  in  the  area  where  station  reliability  is  lower  than  50 
per  cent  due  to  the  extreme  nonlinearities  of  the  reliability  and  cost  functions 
in  this  regime.  (In  the  present  instance  the  reranking  was  extended  back 
from  a-1,  the  approximate  50  per  cent  reliability  point,  to  e-3). 


We  are  now  in  a  position  to  calculate  the  total  program  cost  as  a 
function  of  the  successive  incorporation  of  alternatives.  Table  VIII  illus¬ 
trates  this  calculation  using  equation  (23).  Figure  3  illustrates  the  results 
of  Table  VIE  in  graphic  form  plotted  against  ZW.  Minimum  total  program 
cost  clearly  occurs  at  the  point  that  all  alternatives  up  to  and  including  c-1 
have  been  incorporated.  At  this  point, 


2321  pounds 
1179  pounds 


EM. 

i 


C 

o 


.  0599  man-hours 

.9124 

.7515 

31C.  4  x  10^  dollars 
560.  4  x  106  dollars. 


06) 


Note  that  the  number  of  successful  flights  is  still  time  constrained 
at  this  point.  The  payload  constraint  would  not  take  effect  until  the  next 
alternative  is  incorporated.  Note  that  this  implies  that  flights  somewhat 
longer  than  thirty  days  are  desirable.  This,  of  course,  would  require  a 
change  in  the  original  requirements. 

9.4  Errors  of  Approximation  The  results  which  have  been  obtained 
above  were  based  on  a  variety  of  simplifying  approximations,  the  most 
serious  of  which  was  the  replacement  of  R  *  by  equation  (17c).  The  effect 
of  this  approximation  is  indicated  by  the  dotted  line  in  Figure  2.  Note  that 
agreement  is  good  for  R#  >  0.  75,  but  below  this  figure  there  is  a  rapid 
divergence  between  the  exact  and  approximate  equations.  Fortunately,  this 
has  no  effect  on  our  final  result  since  R^  >  .9  at  the  optimum.  However,  it 


TABLE  VHI 


.  RESULTS  OF  TABLE  VIH  SHOWN  GRAPHICALLY 
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FIGURE  J 


must  be  borne  in  mind  at  all  times  that  the  use  of  approximations  requirec 
that  the  analyst  review  his  results  carefully. 

10.0  INCORPORATION  OF  RxSK  AND  UNCERTAINTY 


Our  discussion  would  not  be  complete  if  we  ignored  the  questions  of 
risk  and  uncertainty.  We  will  dwell  on  this  aspect  of  the  analysis  only  in 
sufficient  detail  to  illustrate  its  importance. 


In  the  present  example  the  uncertainty  aspects  will  evidence  them¬ 
selves  in  estimating  cost,  weight,  and  reliability.  In  view  of  this,  it  is 
unwise  to  interpret  the  scale  of  Figure  3  as  an  absolute  index  of  cost.  On 
the  other  hand,  there  is  considerably  more  certainty  in  the  relative  ranking 
of  the  alternatives  since  errors  made  in  judging  cost,  weight,  and  reliability 
from  one  alternative  to  another  will  tend  to  vary  in  much  the  same  direction. 
Thus  we  may  be  relatively  sure  that  the  optimum  point  of  the  program  cost 
curve  of  Figure  3  represents  the  best  choice  of  alternatives. 


The  question  of  risk  is  synonymous  with  the  question  of  calculable 
odds.  That  is,  the  probability  of  completing  a  given  mission  successfully 
with  the  optimum  system  of  Figure  3  is 


where 


R  P1/4  =  .7515  x  .782 

=  .588 


p 

m  =  probability  that  station  will  be  successfully  orbited  and 
will  remain  nonfailed  for  the  time  duration  L 

tT\ 

P'  *  probability  that  all  fifteen  experiments  will  remain  nonfailed 
pj^4  =  average  experiment  probability  per  flight 

Based  on  this  figure  the  expected  number  of  flights  is 


The  entire  program  can  be  completed  in  (our  totally  euccess(ul 
flight  (N#  =  4).  The  probability  that  thia  will  occur  is 

P  [N  £  4]  *  R4  P 
m  6 

=  (. 7515)4  0.373  =  0.119. 

The  probability  that  it  will  take  exactly  five  flights  to  terminate  the 
program  is  approximated  by 

p  [n  *  53  ~  {(4)  r  3  P3/4  (I  -  R  P1/4)}  R  P*/4 

v  i  m  t  m  t  j  m  e 


ft  R>e/4«l* 


R  Pl/4)2 
m  e 


=  0.334 

The  first  term  of  this  equation  corresponds  to  the  probability  that 
exactly  one  flight  and/or  experiment  failure  occurs  in  the  first  four  flight 
attempts,  and  that  the  fifth  flight  is  an  unqualified  success.  The  second 
term  accounts  for  the  possibility  that  two  successive  flights  do  not  fully 
succeed.  Under  a  ground  rule  established  earlier,  no  experiment  is 
scheduled  to  be  carried  on  more  than  two  successive  flight  attempts;  hence, 
two  successive  flight  failures  (or  partial  successes)  eliminates  any  further 
consideration  of  these  particular  experiments. 

We  shall  bypass  the  question  of  six  flights  temporarily  and  consider 
the  full  impact  of  this  ground  rule.  We  may  interpret  this  as  implying  that 
there  is  a  maximum  number  of  flight  attempts.  Specifically,  if  there  is  a 
succession  of  seven  failures  (or  partial  successes),  the  program  will  ter* 
minate  on  the  eighth  flight  attempt  irrespective  of  the  results  achieved  in 
the  program.  The  probability  that  this  will  occur  is 

P  TN  •  8]  =  (I  -  R  Pl/4)7  =  0.002 
me 

Similarly,  t'ne  oply  way  in  which  the  program  can  terminate  in  exactly 
seven  flight  attempts  is  to  have  a  succession  of  six  failures  (or  partial 
successes)  preceedcd  or  followed  by  a  fully  successful  flight.  Hence, 
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0.006. 


i 


P  [N  =  7]  =  2  (1 

Then  the  probability  that  t 
flight  attempts  it  given  by: 

P  [N  =  6]  =  1  -  P  [N  *  4]  - 
=  0.539 


p”V 

i  e 


R  P 
m 


1/4  , 


agrarn  will  terminate  in  exactly  aix 


P  [N  e  5]  -  P  [N  s  7]  -  P  [N  *  8] 


Under  this  ground  rule,  then,  the  probability  that  the  program  will 
terminate  on  the  N**1  flight  ia  given  by: 

N  PN 

1  0 

2  0 

3  0 

4  0.119 

5  0.453 

6  0.992 

7  0.998 

8  1.000 

Clearly,  thia  ground  rule  makaa  it  easy  to  aay  one  should  buy  aix  stations 
and  take  an  option  on  two  spares.  On  the  other  hand,  this  ground  rule  has 
a  very  inflexible  ring  to  it.  If  the  experiments  are  worth  doing  at  all,  a 
string  of  total  flight  failures  is  unllkel  y  to  cause  abandonment  of  the 
scheduled  experiments.  It  is  more  likely  to  lead  to  rescheduling  anil  a 
total  slippage  of  program  schedule  with  the  inevitable  rise  in  total  program 
cost.  Obviously  a  management  review  of  the  ground  rules  would  be  neces* 
sary  before  over -all  program  cost  could  be  predicted  with  certainty. 

11.0  SUMMARY  AND  CONCLUSION 

A  method  by  means  of  which  it  is  possible  to  seek  a  minimum  cost 
program  has  been  illustrated  in  some  detail.  This  method  is  particularly 
useful  when  there  are  fixed  program  objectives  and  system  effectiveness  is 
not  constrained. 
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»  *■ »  fh-  ala*,  enter  ttu*  aw*'  «  * 

!■»  A  v  4 II  Mill.  ITT  LIMIT  A  IMS  NOTH  f  s  4  >  ;*m- 

m  I  -m  '  hr '  tliiaemma  «  ■  ■■*  t  >  f»c  »*  j*  r*  ’hr*  ‘h**  m 


napoaod  by  aocurtty  cl  see  (fleet  ion.  uaiag  atondord  atataeaontr 
each  a  a: 

(t)  1 'Qualified  re  gw*  alarm  way  ebtaia  rapioo  af  th«a 

raport  Root  DDC." 

(2)  '‘foreign  announcement  and  di  a  aouii  nation  of  this 
(upon  by  DDC  la  not  author* red. •• 

(3)  "U  i.  Gcriraam  agercteo  may  obtain  capiaa  of 
thia  t  sport  directly  from  DDC.  Other  gwaitflad  DOC 
uaaea  ahaJ  1  rogueet  through 


'HI.  1  military  aganciaa  My  obtain  cop  tea  of  ttua 
(upon  directly  Rum  DCC.  Other  qualified  a  aura 
ahall  rogue* t  through 


'All  dietrit  a*.  .**  nr  tlua  raport  la  control  I  ad  Qual¬ 
ified  DENT  wears  ehali  eager  at  through 


If  the  rap  an  ha  a  boon  Ivmehod  ta  Iho  Offkco  of  Technical 
Snetcn.  Dapartmoeu  of  leaarrer,  hr  gala  ta  The  pidlkc.  intfa- 
cata  thia  fact  and  ret*  th*  prtca.  if  la««n 

It  tUPPLOCENTARY  NOTES  Ua*  fur  additional  e^tena- 
tory  noton 

11  tPOfOORING  MtUT ARY  ACTIVITY  Eenar  the  name  of 
iho  dap  on  mo  at  a!  protoe  i  office  at  tabor  Hoc  y  apeaaurtug  fr»r 
<hi  for;  i ha  rgaa arch  and  deialapmaal  Include  rlbtaa 
M  ABSTRACT  Eater  on  abattoir  gtring  a  bn#  I  and  fee  fuel 
•eaaeiy  »l  the  4oc  esmaal  end*  ref!*#  of  the  ripen,  a  von  though  j 
.(  mo*  at***  eppoe»  elaowhara  tm  Mm-  body  af  the  technical  re  , 
port  ll  additicnal  ape*  a  ta  repaired  a  .  aot»n*ai»*n  a  heat  ahall  ■ 

be  «tti<  t>r4  | 

It  »a  highly  dr  a  treble  that  th#  aba(ro*t  o|  <  loaa*fi#4  teporte 
b#  wr  Ia#a*fted  to.  h  paragraph  of  the  abode*  t  ahall  rad  with 
•a  injitaii.m  ol  th#  military  •arunty  <  loaaific  a  (ton  of  the  la 
bmiii.a  in  the  paragrugh  raprvaaotrd  a*  ri»  '1<  *€  >  •»  <tfi 

There  i*  iu  liuitaiioo  on  th#  Irtgtfe  of  th#  •  bat  roc  t  Now 
r  *>e »  *br  *d(|ntfd  length  i«  iron  I  40  •  ■  .'24  w»«da 

14  fcf>  R(>RDS  hey  ••■•d*  are  le.hnna'ly  meonwvgful  tarma  I 
•  *t>  |4t«in  thet  *haia.*eriae  •  r*-p..rt  and  m«y  be  uaed  •• 
iivS*  »  *-«tnr*  Iv  .  ili!.y  >s«  iSe  report  Key  word*  Swil  be 
>*l"  '*1  *  1b#i  .*  >r.  un*t  i  !*»»iK*ti.n  it  rryyirrf  Ideati 

t-.f*  «w  h  •»  ry«i|‘»r*t  mdrl  *> *>|Mtin«  tradr  MtM  military 
•  M*e  n#**.  **<«gtoph..  I  .#•»  ■**  me*  tw  ward  aa  e*y 

•  ■♦.(»  (*.  ’  will  h*-  f  '  I>**e4  t»*  an  iiw..  •(■  m  .•!  «#.  Lu.  »1 
*.  *•  TV*  Mi'rArr"'  .'I  tint*  «u’.r*  an4  weight#  ta  optional 


